Ruby | Support installing the gem via git and some other small build tweaks (#21061)
## Overview
This PR introduces a couple of small tweaks to the Ruby `google-protobuf` gem build. We've confirmed that the packaged `.gem` that gets created as part of `bazel build //ruby:release` is identical to that in the latest `google-protobuf` Ruby gem release. We also had an internal review of these changes, and if it's helpful, that can be found here: https://github.com/Shopify/protobuf/pull/12
## Make the gem installable via git
Ruby `Gemfile`s typically allow you to define a `git` source for gems to easily be able to test custom branches. However, given the following in a `Gemfile`:
```
# Gemfile
gem "google-protobuf", git: "https://github.com/Shopify/protobuf.gif", branch: "master", submodules: true
```
we get the following build output when we try to `bundle install`:
```
Fetching https://github.com/Shopify/protobuf.git
Fetching gem metadata from https://pkgs.shopify.io/basic/gems/ruby/...
Fetching gem metadata from https://rubygems.org/......
Resolving dependencies...
Resolving dependencies...
Gem::Ext::BuildError: ERROR: Failed to build gem native extension.
current directory: /Users/dave/.gem/ruby/3.4.1/bundler/gems/protobuf-8b63023562e0/ruby/ext/google/protobuf_c
/opt/rubies/3.4.1/bin/ruby extconf.rb
creating Makefile
current directory: /Users/dave/.gem/ruby/3.4.1/bundler/gems/protobuf-8b63023562e0/ruby/ext/google/protobuf_c
make DESTDIR\= sitearchdir\=./.gem.20250331-44786-nchu9i sitelibdir\=./.gem.20250331-44786-nchu9i clean
current directory: /Users/dave/.gem/ruby/3.4.1/bundler/gems/protobuf-8b63023562e0/ruby/ext/google/protobuf_c
make DESTDIR\= sitearchdir\=./.gem.20250331-44786-nchu9i sitelibdir\=./.gem.20250331-44786-nchu9i
compiling protobuf.c
In file included from protobuf.c:8:
In file included from ./protobuf.h:23:
In file included from ./defs.h:12:
./ruby-upb.h:1018:18: warning: implicit conversion loses integer precision: 'long' to 'int' [-Wshorten-64-to-32]
1018 | *overrun = ptr - e->end;
| ~ ~~~~^~~~~~~~
./ruby-upb.h:1151:64: warning: implicit conversion loses integer precision: 'size_t' (aka 'unsigned long') to 'int' [-Wshorten-64-to-32]
1151 | upb_EpsCopyInputStream_CheckDataSizeAvailable(e, ptr, size);
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^~~~
./ruby-upb.h:1222:65: warning: implicit conversion loses integer precision: 'size_t' (aka 'unsigned long') to 'int' [-Wshorten-64-to-32]
1222 | if (!upb_EpsCopyInputStream_CheckDataSizeAvailable(e, *ptr, size)) {
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^~~~
./ruby-upb.h:1228:66: warning: implicit conversion loses integer precision: 'size_t' (aka 'unsigned long') to 'int' [-Wshorten-64-to-32]
1228 | const char* ret = upb_EpsCopyInputStream_Copy(e, *ptr, data, size);
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^~~~
./ruby-upb.h:1314:27: warning: implicit conversion loses integer precision: 'long' to 'int' [-Wshorten-64-to-32]
1314 | e->limit = e->limit_ptr - e->end;
| ~ ~~~~~~~~~~~~~^~~~~~~~
./ruby-upb.h:1390:19: warning: implicit conversion loses integer precision: 'size_t' (aka 'unsigned long') to 'int' [-Wshorten-64-to-32]
1390 | return a.size - b.size;
| ~~~~~~ ~~~~~~~^~~~~~~~
./ruby-upb.h:4358:19: warning: implicit conversion loses integer precision: 'size_t' (aka 'unsigned long') to 'int' [-Wshorten-64-to-32]
4357 | array = UPB_PRIVATE(_upb_Array_New)(
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~
4358 | arena, 4, UPB_PRIVATE(_upb_MiniTableField_ElemSizeLg2)(f));
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
./ruby-upb.h:275:24: note: expanded from macro 'UPB_PRIVATE'
275 | #define UPB_PRIVATE(x) x##_dont_copy_me__upb_internal_use_only
| ^
<scratch space>:19:1: note: expanded from here
19 | _upb_MiniTableField_ElemSizeLg2_dont_copy_me__upb_internal_use_only
| ^
In file included from protobuf.c:8:
In file included from ./protobuf.h:23:
In file included from ./defs.h:12:
./ruby-upb.h:14850:10: warning: implicit conversion loses integer precision: 'uint64_t' (aka 'unsigned long long') to 'uint32_t' (aka 'unsigned int') [-Wshorten-64-to-32]
14850 | *tag = val;
| ~ ^~~
./ruby-upb.h:14886:11: warning: implicit conversion loses integer precision: 'uint64_t' (aka 'unsigned long long') to 'int' [-Wshorten-64-to-32]
14886 | *size = size64;
| ~ ^~~~~~
./ruby-upb.h:15299:10: fatal error: 'utf8_range.h' file not found
15299 | #include "utf8_range.h"
| ^~~~~~~~~~~~~~
9 warnings and 1 error generated.
make: *** [protobuf.o] Error 1
make failed, exit code 2
Gem files will remain installed in /Users/dave/.gem/ruby/3.4.1/bundler/gems/protobuf-8b63023562e0/ruby for inspection.
Results logged to /Users/dave/.gem/ruby/3.4.1/bundler/gems/extensions/arm64-darwin-23/3.4.0/protobuf-8b63023562e0-ruby/gem_make.out
/opt/rubies/3.4.1/lib/ruby/3.4.0/rubygems/ext/builder.rb:126:in 'Gem::Ext::Builder.run'
/opt/rubies/3.4.1/lib/ruby/3.4.0/rubygems/ext/builder.rb:52:in 'block in Gem::Ext::Builder.make'
/opt/rubies/3.4.1/lib/ruby/3.4.0/rubygems/ext/builder.rb:44:in 'Array#each'
/opt/rubies/3.4.1/lib/ruby/3.4.0/rubygems/ext/builder.rb:44:in 'Gem::Ext::Builder.make'
/opt/rubies/3.4.1/lib/ruby/3.4.0/rubygems/ext/ext_conf_builder.rb:44:in 'Gem::Ext::ExtConfBuilder.build'
/opt/rubies/3.4.1/lib/ruby/3.4.0/rubygems/ext/builder.rb:195:in 'Gem::Ext::Builder#build_extension'
/opt/rubies/3.4.1/lib/ruby/3.4.0/rubygems/ext/builder.rb:229:in 'block in Gem::Ext::Builder#build_extensions'
/opt/rubies/3.4.1/lib/ruby/3.4.0/rubygems/ext/builder.rb:226:in 'Array#each'
/opt/rubies/3.4.1/lib/ruby/3.4.0/rubygems/ext/builder.rb:226:in 'Gem::Ext::Builder#build_extensions'
/opt/rubies/3.4.1/lib/ruby/3.4.0/rubygems/installer.rb:844:in 'Gem::Installer#build_extensions'
/Users/dave/.gem/ruby/3.4.1/gems/bundler-2.6.3/lib/bundler/rubygems_gem_installer.rb:111:in 'Bundler::RubyGemsGemInstaller#build_extensions'
/Users/dave/.gem/ruby/3.4.1/gems/bundler-2.6.3/lib/bundler/source/path/installer.rb:28:in 'Bundler::Source::Path::Installer#post_install'
/Users/dave/.gem/ruby/3.4.1/gems/bundler-2.6.3/lib/bundler/source/path.rb:234:in 'Bundler::Source::Path#generate_bin'
/Users/dave/.gem/ruby/3.4.1/gems/bundler-2.6.3/lib/bundler/source/git.rb:212:in 'Bundler::Source::Git#install'
/Users/dave/.gem/ruby/3.4.1/gems/bundler-2.6.3/lib/bundler/installer/gem_installer.rb:55:in 'Bundler::GemInstaller#install'
/Users/dave/.gem/ruby/3.4.1/gems/bundler-2.6.3/lib/bundler/installer/gem_installer.rb:17:in 'Bundler::GemInstaller#install_from_spec'
/Users/dave/.gem/ruby/3.4.1/gems/bundler-2.6.3/lib/bundler/installer/parallel_installer.rb:133:in 'Bundler::ParallelInstaller#do_install'
/Users/dave/.gem/ruby/3.4.1/gems/bundler-2.6.3/lib/bundler/installer/parallel_installer.rb:124:in 'block in Bundler::ParallelInstaller#worker_pool'
/Users/dave/.gem/ruby/3.4.1/gems/bundler-2.6.3/lib/bundler/worker.rb:62:in 'Bundler::Worker#apply_func'
/Users/dave/.gem/ruby/3.4.1/gems/bundler-2.6.3/lib/bundler/worker.rb:57:in 'block in Bundler::Worker#process_queue'
<internal:kernel>:168:in 'Kernel#loop'
/Users/dave/.gem/ruby/3.4.1/gems/bundler-2.6.3/lib/bundler/worker.rb:54:in 'Bundler::Worker#process_queue'
/Users/dave/.gem/ruby/3.4.1/gems/bundler-2.6.3/lib/bundler/worker.rb:90:in 'block (2 levels) in Bundler::Worker#create_threads'
An error occurred while installing google-protobuf (4.31.0), and Bundler cannot continue.
In Gemfile:
grpc was resolved to 1.71.0, which depends on
googleapis-common-protos-types was resolved to 1.19.0, which depends on
google-protobuf
```
This is because building the `protobuf_c` Ruby C extension requires the `copy_third_party` rake task to copy the `third_party/utf8_range` library into `ruby/ext/google/protobuf_c`. Fortunately, the `extensions` array in a `Gemfile` can accept either a relative path to an `extconf.rb`, or a `Rakefile`. In the case of the latter, it runs the default `rake`, which in the case of `google-protobuf` is what we need to successfully build the native bindings. The `bazel` build process takes care of these steps which is why `bazel build //ruby:release` works as expected.
However, this `Rakefile` contains tasks that reference files outside of `ruby/`, which won't exist when installing the packaged `.gem` (the artifact that gets published to rubygems.org). The changes here will conditionally set the relevant `extensions` entry to either `ext/google/protobuf_c/extconf.rb` or `Rakefile` depending on the broader build context. This also required a small tweak in the `Rakefile` to modify the `Gem::PackageTask`.
## Missing `google/protobuf/plugin_pb.rb` in `rake gem` artifact
Another issue we noticed is the compiled `src/google/protobuf/compiler/plugin.proto` is absent in the packaged `.gem` that gets outputted when following the [Ruby development build instructions](https://github.com/protocolbuffers/protobuf/tree/main/ruby#installation-from-source-building-gem). This is because this particular proto file was absent from the `well_known_protos` definition in the `Rakefile`. This adds that missing proto to the list of well known protos and makes a small tweak to ensure it ends up in the gem's `lib/google/protobuf` directory instead of `lib/google/protobuf/compiler`. I'm not entirely sure why that directory tree gets a bit flattened, but we wanted to ensure the output remained consistent. Again, this is another issue that the `bazel` build takes care of.
## Tweak `ruby/BUILD.bazel` to also work on macOS
Finally, this PR introduces a small change to `ruby/BUILD.bazel` so `bazel build //ruby:release` can run on macOS as well as Linux. `cp --parents` is not supported in BSD's it seems.
Closes #21061
COPYBARA_INTEGRATE_REVIEW=https://github.com/protocolbuffers/protobuf/pull/21061 from Shopify:ruby-installable-from-git e6b1f0cdad6a42952cd5aa4f6b185fcebaf50191
PiperOrigin-RevId: 744700849 D
Dave Benvenuti committed
d3560e72e791cb61c24df2a1b35946efbd972738
Parent: 4ce6d36
Committed by Copybara-Service <copybara-worker@google.com>
on 4/7/2025, 1:31:44 PM