java.lang.VerifyError：google-adsAPI和SBT的運算元堆疊溢位

我正在嘗試從 Google-AdWords 遷移到 EMR 中 spark 3.1.1 中的 google-ads-v10 API。由于與現有 jar 的沖突，我面臨一些依賴問題。最初，我們面臨與 Protobuf jar 相關的依賴項：

Exception in thread "grpc-default-executor-0" java.lang.IllegalAccessError: tried to access field com.google.protobuf.AbstractMessage.memoizedSize from class com.google.ads.googleads.v10.services.SearchGoogleAdsRequest
    at com.google.ads.googleads.v10.services.SearchGoogleAdsRequest.getSerializedSize(SearchGoogleAdsRequest.java:394)
    at io.grpc.protobuf.lite.ProtoInputStream.available(ProtoInputStream.java:108)

為了解決這個問題，嘗試遮蔽 Protobuf jar 并改用 uber-jar。著色后，在 IntelliJ 中本地運行我的專案作業正常，但是當嘗試運行我創建的可執行 jar 時，出現以下錯誤：

Exception in thread "main" io.grpc.ManagedChannelProvider$ProviderNotFoundException: No functional channel service provider found. Try adding a dependency on the grpc-okhttp, grpc-netty, or grpc-netty-shaded artifact

我嘗試添加所有這些庫，--spark.jars.packages但沒有幫助。

java.lang.VerifyError: Operand stack overflow
Exception Details:
  Location:
    io/grpc/internal/TransportTracer.getStats()Lio/grpc/InternalChannelz$TransportStats; ...
...
...

    at io.grpc.netty.shaded.io.grpc.netty.NettyChannelBuilder.<init>(NettyChannelBuilder.java:96)
    at io.grpc.netty.shaded.io.grpc.netty.NettyChannelBuilder.forTarget(NettyChannelBuilder.java:169)
    at io.grpc.netty.shaded.io.grpc.netty.NettyChannelBuilder.forAddress(NettyChannelBuilder.java:152)
    at io.grpc.netty.shaded.io.grpc.netty.NettyChannelProvider.builderForAddress(NettyChannelProvider.java:38)
    at io.grpc.netty.shaded.io.grpc.netty.NettyChannelProvider.builderForAddress(NettyChannelProvider.java:24)
    at io.grpc.ManagedChannelBuilder.forAddress(ManagedChannelBuilder.java:39)
    at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createSingleChannel(InstantiatingGrpcChannelProvider.java:348)

有沒有人遇到過這樣的問題？

構建.sbt

lazy val dependencies = new {
  val sparkRedshift = "io.github.spark-redshift-community" %% "spark-redshift" % "5.0.3" % "provided" excludeAll (ExclusionRule(organization = "com.amazonaws"))
  val jsonSimple = "com.googlecode.json-simple" % "json-simple" % "1.1" % "provided"
  val googleAdsLib = "com.google.api-ads" % "google-ads" % "17.0.1"
  val jedis = "redis.clients" % "jedis" % "3.0.1" % "provided"
  val sparkAvro = "org.apache.spark" %% "spark-avro" % sparkVersion % "provided"
  val queryBuilder = "com.itfsw" % "QueryBuilder" % "1.0.4" % "provided" excludeAll (ExclusionRule(organization = "com.fasterxml.jackson.core"))
  val protobufForGoogleAds = "com.google.protobuf" % "protobuf-java" % "3.18.1"
  val guavaForGoogleAds = "com.google.guava" % "guava" % "31.1-jre"
}

libraryDependencies   = Seq(
  dependencies.sparkRedshift, dependencies.jsonSimple, dependencies.googleAdsLib,dependencies.guavaForGoogleAds,dependencies.protobufForGoogleAds
  ,dependencies.jedis, dependencies.sparkAvro,
  dependencies.queryBuilder
)


dependencyOverrides   = Set(
  dependencies.guavaForGoogleAds
)

assemblyShadeRules in assembly := Seq(
  ShadeRule.rename("com.google.protobuf.**" -> "repackaged.protobuf.@1").inAll
)


assemblyMergeStrategy in assembly := {
 case PathList("META-INF", xs@_*) => MergeStrategy.discard
 case PathList("module-info.class", xs@_*) => MergeStrategy.discard
 case x => MergeStrategy.first
}

uj5u.com熱心網友回復：

我有一個類似的問題，我將程式集合并策略更改為：

assemblyMergeStrategy in assembly := {
  case x if x.contains("io.netty.versions.properties") => MergeStrategy.discard
  case x =>
    val oldStrategy = (assemblyMergeStrategy in assembly).value
    oldStrategy(x)
}

uj5u.com熱心網友回復：

通過將google-ads-shadowjarjar 用作外部 jar 而不是依賴于 google-ads 庫來解決此問題。這解決了必須手動處理依賴項的問題，但會使您的 jar 大小更大。

標籤：斯卡拉 阿帕奇火花 sbt 亚马逊-emr 谷歌广告 API

上一篇：Pyspark中的滯后移位功能

下一篇：在pyspark中使用string<array>和string創建一個表