主頁 > 企業開發 > 從臨時表請求資料時，Spark查詢中的記憶體泄漏導致錯誤

從臨時表請求資料時，Spark查詢中的記憶體泄漏導致錯誤

2021-12-26 23:02:59 企業開發

首先，我在scala代碼中將資料添加到tempTable中：

resultIndexed.show(490, false)    
resultIndexed.registerTempTable("pivoted")

然后讀入python（省略匯入）：

%spark.pyspark

import bkzep
import numpy as np
from bokeh.io import output_notebook, show
from bokeh.plotting import figure
from bokeh.models import ColumnDataSource
from bokeh.layouts import gridplot
from pyspark.sql.functions import col, coalesce, lit, monotonically_increasing_id
from pyspark.sql import DataFrame
from pyspark.sql.functions import *

output_notebook(notebook_type='zeppelin')


%pyspark

from pyspark.sql.functions import *

def plot_summaries(sensor, dfName):
    df = sqlContext.table(dfName)
    pdf = df.toPandas()
      
        

    #sample = pdf.sample(50)
    source = ColumnDataSource(pdf)
    #print(pdf)

    TOOLTIPS = [    
        ("month", "@month"),        
        ("day", "@day"),    
        ("hour", "@hour"),
        ("min", "@{min}{0.3f}"),
        ("avg", "@{avg}{0.3f}"),
        ("max", "@{max}{0.3f}"),
        ("median", "@{median}{0.3f}"),
        ("stddev", "@{stddev}{0.3f}"),    
    ]

    TOOLTIPS2 = [    
        ("month", "@month"),
        ("day", "@day"),    
        ("count", "@{count}{0.3f}"),    
    ]

    fig = figure(title="Hourly summaries of '{}'".format(sensor),  tooltips=TOOLTIPS)
    #fig.line(x='id', y='avg', source=source, color="orange")
    #fig.line(x='id', y='min', source=source, color="green")
    #fig.line(x='id', y='max', source=source, color="red")
    fig.line(x='id', y='median', source=source, color="blue")
    #fig.line(x='id', y='stddev', source=source, color="aquamarine")

    #fig2 = figure(title="Hourly summaries of '{}' counters".format(sensor),  tooltips=TOOLTIPS2)
    #fig2.line(x='id', y='count', source=source, color="orange")

    show(gridplot([fig], ncols=1, plot_width=1000, plot_height=400))
    #show(fig)

sensors = [
    "Water_Level_Sensor_stddev",
"Water_Level_Sensor_mean"
]


%pyspark

from pyspark.sql.functions import *

keyCol = "month_day_hour"

#for sensor in sensors:
plot_summaries("Water_Level_Sensor_stddev", "pivoted")

然后我在執行程序中收到錯誤：

/spark/python/lib/pyspark.zip/pyspark/sql/pandas/conversion.py:134: UserWarning: toPandas attempted Arrow optimization because 'spark.sql.execution.arrow.pyspark.enabled' is set to true, but has reached the error below and can not continue. Note that 'spark.sql.execution.arrow.pyspark.fallback.enabled' does not have an effect on failures in the middle of computation.
  An error occurred while calling o117.getResult.
: org.apache.spark.SparkException: Exception thrown in awaitResult: 
    at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:302)
    at org.apache.spark.security.SocketAuthServer.getResult(SocketAuthServer.scala:88)
    at org.apache.spark.security.SocketAuthServer.getResult(SocketAuthServer.scala:84)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
    at py4j.Gateway.invoke(Gateway.java:282)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:238)
    at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1157.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1157.0 (TID 81153, 10.244.1.26, executor 12): org.apache.spark.util.TaskCompletionListenerException: Memory was leaked by query. Memory leaked: (671744)
Allocator(toBatchIterator) 0/671744/671744/9223372036854775807 (res/actual/peak/limit)


Previous exception in task: sun.misc.Unsafe or java.nio.DirectByteBuffer.<init>(long, int) not available
    io.netty.util.internal.PlatformDependent.directBuffer(PlatformDependent.java:490)
    io.netty.buffer.NettyArrowBuf.getDirectBuffer(NettyArrowBuf.java:243)
    io.netty.buffer.NettyArrowBuf.nioBuffer(NettyArrowBuf.java:233)
    io.netty.buffer.ArrowBuf.nioBuffer(ArrowBuf.java:245)
    org.apache.arrow.vector.ipc.message.ArrowRecordBatch.computeBodyLength(ArrowRecordBatch.java:222)
    org.apache.arrow.vector.ipc.message.MessageSerializer.serialize(MessageSerializer.java:240)
    org.apache.arrow.vector.ipc.message.MessageSerializer.serialize(MessageSerializer.java:226)
    org.apache.spark.sql.execution.arrow.ArrowConverters$$anon$1.$anonfun$next$1(ArrowConverters.scala:118)
    scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
    org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
    org.apache.spark.sql.execution.arrow.ArrowConverters$$anon$1.next(ArrowConverters.scala:121)
    org.apache.spark.sql.execution.arrow.ArrowConverters$$anon$1.next(ArrowConverters.scala:97)
    scala.collection.Iterator.foreach(Iterator.scala:941)
    scala.collection.Iterator.foreach$(Iterator.scala:941)
    org.apache.spark.sql.execution.arrow.ArrowConverters$$anon$1.foreach(ArrowConverters.scala:97)
    scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:62)
    scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:53)
    scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:105)
    scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:49)
    scala.collection.TraversableOnce.to(TraversableOnce.scala:315)
    scala.collection.TraversableOnce.to$(TraversableOnce.scala:313)
    org.apache.spark.sql.execution.arrow.ArrowConverters$$anon$1.to(ArrowConverters.scala:97)
    scala.collection.TraversableOnce.toBuffer(TraversableOnce.scala:307)
    scala.collection.TraversableOnce.toBuffer$(TraversableOnce.scala:307)
    org.apache.spark.sql.execution.arrow.ArrowConverters$$anon$1.toBuffer(ArrowConverters.scala:97)
    scala.collection.TraversableOnce.toArray(TraversableOnce.scala:294)
    scala.collection.TraversableOnce.toArray$(TraversableOnce.scala:288)
    org.apache.spark.sql.execution.arrow.ArrowConverters$$anon$1.toArray(ArrowConverters.scala:97)
    org.apache.spark.sql.Dataset.$anonfun$collectAsArrowToPython$6(Dataset.scala:3562)
    org.apache.spark.SparkContext.$anonfun$runJob$6(SparkContext.scala:2193)
    org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
    org.apache.spark.scheduler.Task.run(Task.scala:127)
    org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446)
    org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
    org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449)
    java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    java.base/java.lang.Thread.run(Unknown Source)
    at org.apache.spark.TaskContextImpl.invokeListeners(TaskContextImpl.scala:145)
    at org.apache.spark.TaskContextImpl.markTaskCompleted(TaskContextImpl.scala:124)
    at org.apache.spark.scheduler.Task.run(Task.scala:137)
    at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.base/java.lang.Thread.run(Unknown Source)

Driver stacktrace:
    at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2059)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2008)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2007)
    at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
    at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2007)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:973)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:973)
    at scala.Option.foreach(Option.scala:407)
    at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:973)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2239)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2188)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2177)
    at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:775)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2099)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2194)
    at org.apache.spark.sql.Dataset.$anonfun$collectAsArrowToPython$5(Dataset.scala:3560)
    at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
    at org.apache.spark.sql.Dataset.$anonfun$collectAsArrowToPython$2(Dataset.scala:3564)
    at org.apache.spark.sql.Dataset.$anonfun$collectAsArrowToPython$2$adapted(Dataset.scala:3541)
    at org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3618)
    at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:100)
    at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:160)
    at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:87)
    at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:764)
    at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
    at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3616)
    at org.apache.spark.sql.Dataset.$anonfun$collectAsArrowToPython$1(Dataset.scala:3541)
    at org.apache.spark.sql.Dataset.$anonfun$collectAsArrowToPython$1$adapted(Dataset.scala:3540)
    at org.apache.spark.security.SocketAuthServer$.$anonfun$serveToStream$2(SocketAuthServer.scala:130)
    at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
    at org.apache.spark.security.SocketAuthServer$.$anonfun$serveToStream$1(SocketAuthServer.scala:132)
    at org.apache.spark.security.SocketAuthServer$.$anonfun$serveToStream$1$adapted(SocketAuthServer.scala:127)
    at org.apache.spark.security.SocketFuncServer.handleConnection(SocketAuthServer.scala:104)
    at org.apache.spark.security.SocketFuncServer.handleConnection(SocketAuthServer.scala:98)
    at org.apache.spark.security.SocketAuthServer$$anon$1.$anonfun$run$1(SocketAuthServer.scala:60)
    at scala.util.Try$.apply(Try.scala:213)
    at org.apache.spark.security.SocketAuthServer$$anon$1.run(SocketAuthServer.scala:60)
Caused by: org.apache.spark.util.TaskCompletionListenerException: Memory was leaked by query. Memory leaked: (671744)
Allocator(toBatchIterator) 0/671744/671744/9223372036854775807 (res/actual/peak/limit)


Previous exception in task: sun.misc.Unsafe or java.nio.DirectByteBuffer.<init>(long, int) not available
    io.netty.util.internal.PlatformDependent.directBuffer(PlatformDependent.java:490)
    io.netty.buffer.NettyArrowBuf.getDirectBuffer(NettyArrowBuf.java:243)
    io.netty.buffer.NettyArrowBuf.nioBuffer(NettyArrowBuf.java:233)
    io.netty.buffer.ArrowBuf.nioBuffer(ArrowBuf.java:245)
    org.apache.arrow.vector.ipc.message.ArrowRecordBatch.computeBodyLength(ArrowRecordBatch.java:222)
    org.apache.arrow.vector.ipc.message.MessageSerializer.serialize(MessageSerializer.java:240)
    org.apache.arrow.vector.ipc.message.MessageSerializer.serialize(MessageSerializer.java:226)
    org.apache.spark.sql.execution.arrow.ArrowConverters$$anon$1.$anonfun$next$1(ArrowConverters.scala:118)
    scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
    org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
    org.apache.spark.sql.execution.arrow.ArrowConverters$$anon$1.next(ArrowConverters.scala:121)
    org.apache.spark.sql.execution.arrow.ArrowConverters$$anon$1.next(ArrowConverters.scala:97)
    scala.collection.Iterator.foreach(Iterator.scala:941)
    scala.collection.Iterator.foreach$(Iterator.scala:941)
    org.apache.spark.sql.execution.arrow.ArrowConverters$$anon$1.foreach(ArrowConverters.scala:97)
    scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:62)
    scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:53)
    scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:105)
    scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:49)
    scala.collection.TraversableOnce.to(TraversableOnce.scala:315)
    scala.collection.TraversableOnce.to$(TraversableOnce.scala:313)
    org.apache.spark.sql.execution.arrow.ArrowConverters$$anon$1.to(ArrowConverters.scala:97)
    scala.collection.TraversableOnce.toBuffer(TraversableOnce.scala:307)
    scala.collection.TraversableOnce.toBuffer$(TraversableOnce.scala:307)
    org.apache.spark.sql.execution.arrow.ArrowConverters$$anon$1.toBuffer(ArrowConverters.scala:97)
    scala.collection.TraversableOnce.toArray(TraversableOnce.scala:294)
    scala.collection.TraversableOnce.toArray$(TraversableOnce.scala:288)
    org.apache.spark.sql.execution.arrow.ArrowConverters$$anon$1.toArray(ArrowConverters.scala:97)
    org.apache.spark.sql.Dataset.$anonfun$collectAsArrowToPython$6(Dataset.scala:3562)
    org.apache.spark.SparkContext.$anonfun$runJob$6(SparkContext.scala:2193)
    org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
    org.apache.spark.scheduler.Task.run(Task.scala:127)
    org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446)
    org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
    org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449)
    java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    java.lang.Thread.run(Unknown Source)
    at org.apache.spark.TaskContextImpl.invokeListeners(TaskContextImpl.scala:145)
    at org.apache.spark.TaskContextImpl.markTaskCompleted(TaskContextImpl.scala:124)
    at org.apache.spark.scheduler.Task.run(Task.scala:137)
    at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)

  warnings.warn(msg)
---------------------------------------------------------------------------
Py4JJavaError                             Traceback (most recent call last)
<ipython-input-127-51c79da6d64b> in <module>
      4 
      5 #for sensor in sensors:
----> 6 plot_summaries("Water_Level_Sensor_stddev", "pivoted")

<ipython-input-124-70e98889279f> in plot_summaries(sensor, dfName)
      5     df = sqlContext.table(dfName)
      6 
----> 7     pdf = df.toPandas()
      8         #.select("*") \
      9         #.orderBy("index") \

/spark/python/lib/pyspark.zip/pyspark/sql/pandas/conversion.py in toPandas(self)
    106                     # Rename columns to avoid duplicated column names.
    107                     tmp_column_names = ['col_{}'.format(i) for i in range(len(self.columns))]
--> 108                     batches = self.toDF(*tmp_column_names)._collect_as_arrow()
    109                     if len(batches) > 0:
    110                         table = pyarrow.Table.from_batches(batches)

/spark/python/lib/pyspark.zip/pyspark/sql/pandas/conversion.py in _collect_as_arrow(self)
    242         finally:
    243             # Join serving thread and raise any exceptions from collectAsArrowToPython
--> 244             jsocket_auth_server.getResult()
    245 
    246         # Separate RecordBatches from batch order indices in results

/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py in __call__(self, *args)
   1303         answer = self.gateway_client.send_command(command)
   1304         return_value = get_return_value(
-> 1305             answer, self.gateway_client, self.target_id, self.name)
   1306 
   1307         for temp_arg in temp_args:

/spark/python/lib/pyspark.zip/pyspark/sql/utils.py in deco(*a, **kw)
    126     def deco(*a, **kw):
    127         try:
--> 128             return f(*a, **kw)
    129         except py4j.protocol.Py4JJavaError as e:
    130             converted = convert_exception(e.java_exception)

/spark/python/lib/py4j-0.10.9-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
    326                 raise Py4JJavaError(
    327                     "An error occurred while calling {0}{1}{2}.\n".
--> 328                     format(target_id, ".", name), value)
    329             else:
    330                 raise Py4JError(

Py4JJavaError: An error occurred while calling o117.getResult.
: org.apache.spark.SparkException: Exception thrown in awaitResult: 
    at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:302)
    at org.apache.spark.security.SocketAuthServer.getResult(SocketAuthServer.scala:88)
    at org.apache.spark.security.SocketAuthServer.getResult(SocketAuthServer.scala:84)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
    at py4j.Gateway.invoke(Gateway.java:282)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:238)
    at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1157.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1157.0 (TID 81153, 10.244.1.26, executor 12): org.apache.spark.util.TaskCompletionListenerException: Memory was leaked by query. Memory leaked: (671744)
Allocator(toBatchIterator) 0/671744/671744/9223372036854775807 (res/actual/peak/limit)


Previous exception in task: sun.misc.Unsafe or java.nio.DirectByteBuffer.<init>(long, int) not available
    io.netty.util.internal.PlatformDependent.directBuffer(PlatformDependent.java:490)
    io.netty.buffer.NettyArrowBuf.getDirectBuffer(NettyArrowBuf.java:243)
    io.netty.buffer.NettyArrowBuf.nioBuffer(NettyArrowBuf.java:233)
    io.netty.buffer.ArrowBuf.nioBuffer(ArrowBuf.java:245)
    org.apache.arrow.vector.ipc.message.ArrowRecordBatch.computeBodyLength(ArrowRecordBatch.java:222)
    org.apache.arrow.vector.ipc.message.MessageSerializer.serialize(MessageSerializer.java:240)
    org.apache.arrow.vector.ipc.message.MessageSerializer.serialize(MessageSerializer.java:226)
    org.apache.spark.sql.execution.arrow.ArrowConverters$$anon$1.$anonfun$next$1(ArrowConverters.scala:118)
    scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
    org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
    org.apache.spark.sql.execution.arrow.ArrowConverters$$anon$1.next(ArrowConverters.scala:121)
    org.apache.spark.sql.execution.arrow.ArrowConverters$$anon$1.next(ArrowConverters.scala:97)
    scala.collection.Iterator.foreach(Iterator.scala:941)
    scala.collection.Iterator.foreach$(Iterator.scala:941)
    org.apache.spark.sql.execution.arrow.ArrowConverters$$anon$1.foreach(ArrowConverters.scala:97)
    scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:62)
    scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:53)
    scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:105)
    scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:49)
    scala.collection.TraversableOnce.to(TraversableOnce.scala:315)
    scala.collection.TraversableOnce.to$(TraversableOnce.scala:313)
    org.apache.spark.sql.execution.arrow.ArrowConverters$$anon$1.to(ArrowConverters.scala:97)
    scala.collection.TraversableOnce.toBuffer(TraversableOnce.scala:307)
    scala.collection.TraversableOnce.toBuffer$(TraversableOnce.scala:307)
    org.apache.spark.sql.execution.arrow.ArrowConverters$$anon$1.toBuffer(ArrowConverters.scala:97)
    scala.collection.TraversableOnce.toArray(TraversableOnce.scala:294)
    scala.collection.TraversableOnce.toArray$(TraversableOnce.scala:288)
    org.apache.spark.sql.execution.arrow.ArrowConverters$$anon$1.toArray(ArrowConverters.scala:97)
    org.apache.spark.sql.Dataset.$anonfun$collectAsArrowToPython$6(Dataset.scala:3562)
    org.apache.spark.SparkContext.$anonfun$runJob$6(SparkContext.scala:2193)
    org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
    org.apache.spark.scheduler.Task.run(Task.scala:127)
    org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446)
    org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
    org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449)
    java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    java.base/java.lang.Thread.run(Unknown Source)
    at org.apache.spark.TaskContextImpl.invokeListeners(TaskContextImpl.scala:145)
    at org.apache.spark.TaskContextImpl.markTaskCompleted(TaskContextImpl.scala:124)
    at org.apache.spark.scheduler.Task.run(Task.scala:137)
    at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.base/java.lang.Thread.run(Unknown Source)

Driver stacktrace:
    at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2059)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2008)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2007)
    at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
    at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2007)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:973)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:973)
    at scala.Option.foreach(Option.scala:407)
    at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:973)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2239)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2188)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2177)
    at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:775)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2099)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2194)
    at org.apache.spark.sql.Dataset.$anonfun$collectAsArrowToPython$5(Dataset.scala:3560)
    at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
    at org.apache.spark.sql.Dataset.$anonfun$collectAsArrowToPython$2(Dataset.scala:3564)
    at org.apache.spark.sql.Dataset.$anonfun$collectAsArrowToPython$2$adapted(Dataset.scala:3541)
    at org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3618)
    at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:100)
    at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:160)
    at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:87)
    at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:764)
    at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
    at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3616)
    at org.apache.spark.sql.Dataset.$anonfun$collectAsArrowToPython$1(Dataset.scala:3541)
    at org.apache.spark.sql.Dataset.$anonfun$collectAsArrowToPython$1$adapted(Dataset.scala:3540)
    at org.apache.spark.security.SocketAuthServer$.$anonfun$serveToStream$2(SocketAuthServer.scala:130)
    at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
    at org.apache.spark.security.SocketAuthServer$.$anonfun$serveToStream$1(SocketAuthServer.scala:132)
    at org.apache.spark.security.SocketAuthServer$.$anonfun$serveToStream$1$adapted(SocketAuthServer.scala:127)
    at org.apache.spark.security.SocketFuncServer.handleConnection(SocketAuthServer.scala:104)
    at org.apache.spark.security.SocketFuncServer.handleConnection(SocketAuthServer.scala:98)
    at org.apache.spark.security.SocketAuthServer$$anon$1.$anonfun$run$1(SocketAuthServer.scala:60)
    at scala.util.Try$.apply(Try.scala:213)
    at org.apache.spark.security.SocketAuthServer$$anon$1.run(SocketAuthServer.scala:60)
Caused by: org.apache.spark.util.TaskCompletionListenerException: Memory was leaked by query. Memory leaked: (671744)
Allocator(toBatchIterator) 0/671744/671744/9223372036854775807 (res/actual/peak/limit)

我怎樣才能避免它？

uj5u.com熱心網友回復：

從臨時表請求資料時，Spark 查詢中的記憶體泄漏導致錯誤

但最終在重啟 Spark 解釋器后問題得到解決

轉載請註明出處，本文鏈接：https://www.uj5u.com/qiye/393590.html

標籤：Python 斯卡拉 阿帕奇火花 火花 阿帕奇齐柏林飞艇

上一篇：AkkaHttp路由測驗與formFields導致UnsupportedRequestContentTypeRejection

下一篇：在ApacheZeppelin中使用pyspark讀取DataFrame時出現問題：缺少物件中的某些方法的例外不清楚

標籤雲
其他(157675) Python(38076) JavaScript(25376) Java(17977) C(15215) 區塊鏈(8255) C＃(7972) AI(7469) 爪哇(7425) MySQL(7132) html(6777) 基礎類(6313) sql(6102) 熊猫(6058) PHP(5869) 数组(5741) R(5409) Linux(5327) 反应(5209) 腳本語言(PerlPython)(5129) 非技術區(4971) Android(4554) 数据框(4311) css(4259) 节点.js(4032) C語言(3288) json(3245) 列表(3129) 扑(3119) C++語言(3117) 安卓(2998) 打字稿(2995) VBA(2789) Java相關(2746) 疑難問題(2699) 细绳(2522) 單片機工控(2479) iOS(2429) ASP.NET(2402) MongoDB(2323) 麻木的(2285) 正则表达式(2254) 字典(2211) 循环(2198) 迅速(2185) 擅长(2169) 镖(2155) 功能(1967) .NET技术(1958) Web開發(1951) python-3.x(1918) HtmlCss(1915) 弹簧靴(1913) C++(1909) xml(1889) PostgreSQL(1872) .NETCore(1853) 谷歌表格(1846) Unity3D(1843) for循环(1842)

熱門瀏覽
  • IEEE1588PTP在數字化變電站時鐘同步方面的應用

    IEEE1588ptp在數字化變電站時鐘同步方面的應用 京準電子科技官微——ahjzsz 一、電力系統時間同步基本概況 隨著對IEC 61850標準研究的不斷深入，國內外學者提出基于IEC61850通信標準體系建設數字化變電站的發展思路。數字化變電站與常規變電站的顯著區別在于程序層傳統的電流/電壓互 ......

    uj5u.com 2020-09-10 03:51:52 more
  • HTTP request smuggling CL.TE

    CL.TE 簡介 前端通過Content-Length處理請求，通過反向代理或者負載均衡將請求轉發到后端，后端Transfer-Encoding優先級較高，以TE處理請求造成安全問題。 檢測 發送如下資料包 POST / HTTP/1.1 Host: ac391f7e1e9af821806e890 ......

    uj5u.com 2020-09-10 03:52:11 more
  • 網路滲透資料大全單——漏洞庫篇

    網路滲透資料大全單——漏洞庫篇漏洞庫 NVD ——美國國家漏洞庫 →http://nvd.nist.gov/。 CERT ——美國國家應急回應中心 →https://www.us-cert.gov/ OSVDB ——開源漏洞庫 →http://osvdb.org Bugtraq ——賽門鐵克 →ht ......

    uj5u.com 2020-09-10 03:52:15 more
  • 京準講述NTP時鐘服務器應用及原理

    京準講述NTP時鐘服務器應用及原理京準講述NTP時鐘服務器應用及原理 安徽京準電子科技官微——ahjzsz 北斗授時原理 授時是指接識訓通過某種方式獲得本地時間與北斗標準時間的鐘差，然后調整本地時鐘使時差控制在一定的精度范圍內。 衛星導航系統通常由三部分組成：導航授時衛星、地面檢測校正維護系統和用戶 ......

    uj5u.com 2020-09-10 03:52:25 more
  • 利用北斗衛星系統設計NTP網路時間服務器

    利用北斗衛星系統設計NTP網路時間服務器 利用北斗衛星系統設計NTP網路時間服務器 安徽京準電子科技官微——ahjzsz 概述 NTP網路時間服務器是一款支持NTP和SNTP網路時間同步協議，高精度、大容量、高品質的高科技時鐘產品。 NTP網路時間服務器設備采用冗余架構設計，高精度時鐘直接來源于北斗 ......

    uj5u.com 2020-09-10 03:52:35 more
  • 詳細解讀電力系統各種對時方式

    詳細解讀電力系統各種對時方式 詳細解讀電力系統各種對時方式 安徽京準電子科技官微——ahjzsz，更多資料請添加VX 衛星同步時鐘是我京準公司開發研制的應用衛星授時時技術的標準時間顯示和發送的裝置，該裝置以M國全球定位系統（GLOBAL POSITIONING SYSTEM,縮寫為GPS）或者我國北 ......

    uj5u.com 2020-09-10 03:52:45 more
  • 如何保證外包團隊接入企業內網安全

    不管企業規模的大小，只要企業想省錢，那么企業的某些服務就一定會采用外包的形式，然而看似美好又經濟的策略，其實也有不好的一面。下面我通過安全的角度來聊聊使用外包團的安全隱患問題。 先看看什么服務會使用外包的，最常見的就是話務/客服這種需要大量重復性、無技術性的服務，或者是一些銷售外包、特殊的職能外包等 ......

    uj5u.com 2020-09-10 03:52:57 more
  • PHP漏洞之【整型數字型SQL注入】

    0x01 什么是SQL注入 SQL是一種注入攻擊，通過前端帶入后端資料庫進行惡意的SQL陳述句查詢。 0x02 SQL整型注入原理 SQL注入一般發生在動態網站URL地址里，當然也會發生在其它地發，如登錄框等等也會存在注入，只要是和資料庫打交道的地方都有可能存在。 如這里http://192.168. ......

    uj5u.com 2020-09-10 03:55:40 more
  • [GXYCTF2019]禁止套娃

    git泄露獲取原始碼 使用GET傳參，引數為exp 經過三層過濾執行 第一層過濾偽協議，第二層過濾帶引數的函式，第三層過濾一些函式 preg_replace('/[a-z,_]+\((?R)?\)/', NULL, $_GET['exp'] (?R)參考當前正則運算式，相當于匹配函式里的引數 因此傳遞 ......

    uj5u.com 2020-09-10 03:56:07 more
  • 等保2.0實施流程

    流程 結論 ......

    uj5u.com 2020-09-10 03:56:16 more
最新发布
  • 使用Django Rest framework搭建Blog

    在前面的Blog例子中我們使用的是GraphQL, 雖然GraphQL的使用處于上升趨勢,但是Rest API還是使用的更廣泛一些. 所以還是決定回到傳統的rest api framework上來, Django rest framework的官網上給了一個很好用的QuickStart, 我參考Qu ......

    uj5u.com 2023-04-20 08:17:54 more
  • 記錄-new Date() 我忍你很久了！

    這里給大家分享我在網上總結出來的一些知識，希望對大家有所幫助 大家平時在開發的時候有沒被new Date()折磨過？就是它的諸多怪異的設定讓你每每用的時候，都可能不小心踩坑。造成程式意外出錯，卻一下子找不到問題出處，那叫一個煩透了…… 下面，我就列舉它的“四宗罪”及應用思考 可惡的四宗罪 1. Sa ......

    uj5u.com 2023-04-20 08:17:47 more
  • 使用Vue.js實作文字跑馬燈效果

    實作文字跑馬燈效果，首先用到 substring()截取 和 setInterval計時器 clearInterval()清除計時器 效果如下： 實作代碼如下： <!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <meta ......

    uj5u.com 2023-04-20 08:12:31 more
  • JavaScript 運算子

    JavaScript 運算子/運算子 在 JavaScript 中，有一些運算子可以使代碼更簡潔、易讀和高效。以下是一些常見的運算子： 1、可選鏈運算子（optional chaining operator） ?.是可選鏈運算子（optional chaining operator）。?. 可選鏈操 ......

    uj5u.com 2023-04-20 08:02:25 more
  • CSS—相對單位rem

    一、概述 rem是一個相對長度單位，它的單位長度取決于根標簽html的字體尺寸。rem即root em的意思，中文翻譯為根em。瀏覽器的文本尺寸一般默認為16px，即默認情況下： 1rem = 16px rem布局原理：根據CSS媒體查詢功能，更改根標簽的字體尺寸，實作rem單位隨螢屏尺寸的變化，如 ......

    uj5u.com 2023-04-20 08:02:21 more
  • 我的第一個NPM包:panghu-planebattle-esm(胖虎飛機大戰)使用說明

    好家伙,我的包終于開發完啦 歡迎使用胖虎的飛機大戰包!! 為你的主頁添加色彩 這是一個有趣的網頁小游戲包,使用canvas和js開發 使用ES6模塊化開發 效果圖如下: (覺得圖片太sb的可以自己改) 代碼已開源!! Git: https://gitee.com/tang-and-han-dynas ......

    uj5u.com 2023-04-20 08:01:50 more
  • 如何在 vue3 中使用 jsx/tsx?

    我們都知道,通常情況下我們使用 vue 大多都是用的 SFC(Signle File Component)單檔案組件模式,即一個組件就是一個檔案,但其實 Vue 也是支持使用 JSX 來撰寫組件的。這里不討論 SFC 和 JSX 的好壞,這個仁者見仁智者見智。本篇文章旨在帶領大家快速了解和使用 Vu ......

    uj5u.com 2023-04-20 08:01:37 more
  • 【Vue2.x原始碼系列06】計算屬性computed原理

    本章目標：計算屬性是如何實作的？計算屬性快取原理以及洋蔥模型的應用？在初始化Vue實體時，我們會給每個計算屬性都創建一個對應watcher，我們稱之為計算屬性watcher ......

    uj5u.com 2023-04-20 08:01:31 more
  • http1.1與http2.0

    一、http是什么 通俗來講，http就是計算機通過網路進行通信的規則，是一個基于請求與回應，無狀態的，應用層協議。常用于TCP/IP協議傳輸資料。目前任何終端之間任何一種通信方式都必須按Http協議進行，否則無法連接。tcp(三次握手，四次揮手)。 請求與回應：客戶端請求、服務端回應資料。 無狀態 ......

    uj5u.com 2023-04-20 08:01:10 more
  • http1.1與http2.0

    一、http是什么 通俗來講，http就是計算機通過網路進行通信的規則，是一個基于請求與回應，無狀態的，應用層協議。常用于TCP/IP協議傳輸資料。目前任何終端之間任何一種通信方式都必須按Http協議進行，否則無法連接。tcp(三次握手，四次揮手)。 請求與回應：客戶端請求、服務端回應資料。 無狀態 ......

    uj5u.com 2023-04-20 08:00:32 more

Copyright © 2010-2020 有解無憂