Jay Taylor | programmer notes

Archive for June 2016

# Shut down both VMs.
VBoxManage controlvm gw-lab_mesos-primary1a poweroff
VBoxManage controlvm gw-lab_mesos-primary2a poweroff

# Add a SATA controller port to the target VM (the one where fsck will be run from).
VBoxManage storageattach gw-lab_mesos-primary2a --medium none --storagectl SATAController --port 1 --device 0 --type hdd

# Attach the other hard drive to the target VM.
VBoxManage storageattach gw-lab_mesos-primary2a --medium /mnt/VirtualBox\ VMs/gw-lab_mesos-primary1a/Snapshots/\{4695a86f-e9f3-4e4f-8b48-0336af217815\}.vmdk --storagectl SATAController --port 1 --device 0 --type hdd

# Start the target VM.
VBoxManage startvm --type headless gw-lab_mesos-primary2a

ssh mesos-primary2a sudo fsck /dev/sdb1
y
y
y
y
y
...

Note: At first I somehow managed to attach the drive the mesos-primary2a, such that it showed up in `showhdinfo` but it wasn’t available in the target VM, and couldn’t be removed. Rebooting the host got VBox out of the funky state.

jaytaylor@host:/mnt/VirtualBox VMs$ VBoxManage showhdinfo /mnt/VirtualBox\ VMs/gw-lab_mesos-primary1a/Snapshots/\{4695a86f-e9f3-4e4f-8b48-0336af217815\}.vmdk
UUID: 50d87b4c-2c8d-40df-aeba-2153cbb7066d
Parent UUID: base
State: created
Type: normal (base)
Location: /mnt/VirtualBox VMs/gw-lab_mesos-primary1a/Snapshots/{4695a86f-e9f3-4e4f-8b48-0336af217815}.vmdk
Storage format: VMDK
Format variant: dynamic default
Capacity: 40960 MBytes
Size on disk: 38072 MBytes
In use by VMs: gw-lab_mesos-primary1a (UUID: 2160cfb5-1b5b-4f32-81bf-385f3d7a796a)
gw-lab_mesos-primary2a (UUID: c7a80492-cc66-4460-9b5a-53572875653c)

jaytaylor@host:/mnt/VirtualBox VMs$ VBoxManage showvminfo c7a80492-cc66-4460-9b5a-53572875653c --details
Name: gw-lab_mesos-primary2a
...
Default Frontend:
Storage Controller Name (0): SATAController
Storage Controller Type (0): IntelAhci
Storage Controller Instance Number (0): 0
Storage Controller Max Port Count (0): 30
Storage Controller Port Count (0): 2
Storage Controller Bootable (0): on
SATAController (0, 0): /mnt/VirtualBox VMs/gw-lab_mesos-primary2a/Snapshots/{7149016e-a75d-4612-b63e-52c8c5e45ad8}.vmdk (UUID: 64eb88f4-47cc-43b9-997a-6b1d440015da)
NIC 1: MAC: 0800278FD4DC, Attachment: NAT, Cable connected: on, Trace: off (file: none), Type: 82540EM, Reported speed: 0 Mbps, Boot priority: 0, Promisc Policy: deny, Bandwidth group: none
...

No tags Hide

I needed the binary “grpc_python_plugin” to follow the Python gRPC tutorial.

I’ve hit quite a few snags.

And it appears I’m not the only one

pip install grpio-tools

...

grpc/tools/main.cc:33:10: fatal error: 'src/compiler/python_generator.h' file not found

And the grpc docs don’t include macOS instructions.

Let’s start hacking:

wget https://pypi.python.org/packages/7b/22/93b83676787ab07fb7f8d8dcea5351efd6ee62ca0dfba8799cc06f375b37/grpcio_tools-0.14.0.tar.gz#md5=18dd40dd0ffba48bbb8ab865b7fbd23a

tar zxvf grpcio_tools-0.14.0.tar.gz

cd grpcio_tools

I found the python_generator.h file at https://github.com/grpc/grpc/blob/master/src/compiler/python_generator.h, so:

git clone https://github.com/grpc/grpc grpc_root

python setup.py build

...

clang -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Qunused-arguments -Qunused-arguments -DHAVE_PTHREAD=1 -I. -Igrpc_root -Igrpc_root/include -Ithird_party/protobuf/src -I/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c grpc/tools/main.cc -o build/temp.macosx-10.9-x86_64-2.7/grpc/tools/main.o -frtti -std=c++11
clang -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Qunused-arguments -Qunused-arguments -DHAVE_PTHREAD=1 -I. -Igrpc_root -Igrpc_root/include -Ithird_party/protobuf/src -I/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c grpc_root/src/compiler/python_generator.cc -o build/temp.macosx-10.9-x86_64-2.7/grpc_root/src/compiler/python_generator.o -frtti -std=c++11
clang -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Qunused-arguments -Qunused-arguments -DHAVE_PTHREAD=1 -I. -Igrpc_root -Igrpc_root/include -Ithird_party/protobuf/src -I/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c third_party/protobuf/src/google/protobuf/compiler/zip_writer.cc -o build/temp.macosx-10.9-x86_64-2.7/third_party/protobuf/src/google/protobuf/compiler/zip_writer.o -frtti -std=c++11
clang: error: no such file or directory: 'third_party/protobuf/src/google/protobuf/compiler/zip_writer.cc'
clang: error: no input files
error: command 'clang' failed with exit status 1

clang: error: no such file or directory: 'third_party/protobuf/src/google/protobuf/compiler/zip_writer.cc'

Okay, what a mess! Well okay, I found the set of files in question.

mkdir tmp
cd tmp
wget https://android.googlesource.com/platform/external/chromium_org/+archive/00d67fb/third_party/protobuf/src/google/protobuf.tar.gz
tar xzvf protobuf.tar.gz
rm protobuf.tar.gz
mkdir -p ../grpc_root/third_party/protobuf/src/google/protobuf
mv * ../grpc_root/third_party/protobuf/src/google/protobuf
cd ..

Sadly, even after locating the files and lovingly injecting them, it has no effect and the build still errors out with the same error.

Okay, it turns out that was all wrong. The zip_writer.cc is included with the main grpc repository, it just comes from a submodule.

Let’s try just building that:

GRPC_PYTHON_BUILD_WITH_CYTHON=1 pip install .
...
commands.CommandError: could not find grpc_python_plugin (protoc plugin for GRPC Python)

Reviewing the relevant github issue #5378 grpc_python_plugin is not included with pip install grpcio, it became clear that @revantk hit the exact same problem and set of errors.

Here is my final solution:

Just in case it helps someone else..

If you're missing the `grpc_python_plugin` binary on macOS (Mac OS X?):

git clone https://github.com/google/protobuf.git
cd protobuf
./autogen.sh
./configure
make
make install
cd ..

Then:

git clone https://github.com/grpc/grpc
cd grpc
git submodule update --init --recursive
make grpc_python_plugin
cp bins/opt/grpc_python_plugin /usr/local/bin/

After this I was good to go!

No tags Hide

Frozen Virtual Machines

Lately one of my testing lab Ubuntu Linux hosts has been hanging and/or freezing (requiring a hard system reset) when load was introduced to any of the guest VMs.

A bit of research revealed VBox Ticket #8511: “Regular crashes or freezing”:

One of the most important answers is right there - if you're on a
Linux host and doing heavy disk I/O, do not use the host cache
for the VMs, ever. The Linux I/O subsystem not very smart, it
batches gobs of dirty pages in the filesystem cache, and when it
runs out of free memory, flushes out everything to disk. That can
take quite a long time (minutes) and there's nothing VirtualBox
can do about it.

The asynchronous I/O in VirtualBox was designed explicitly to work
around this host OS deficiency. The I/O doesn't go through the
host's cache and is written to disk much more frequently in smaller
chunks. However, VirtualBox isn't necessarily the only process
running on the host and something else still may trigger the
undesirable behavior.

The corollary to the above is obvious: If your host can't cope with
the I/O load generated by the VMs plus the rest of the system,
there will be trouble. Virtualization isn't magic and can't turn a slow
disk into a fast one.

The operative portion being:

If on a Linux host ... do not use the host cache for the VMs, ever.

Digging into the documentation I found out how to disabled host- caching on a per-VM-controlller basis:

VBoxManage storagectl VM-NAME-HERE --name SATAController --hostiocache off

After applying that to all VMs, voila! All fixed!

They may become slow under heavy load (still better than the freeze ups in the past).

No tags Hide

Find it!