問題發現
早上過來，飯都沒來的及吃，運維就給我發來資訊，說是某個介面呼叫大量超時，因為最近這個介面呼叫量是翻倍了，所以我就去檢查了下慢SQL，發現確實是有較多的慢SQL，所以我就縮減了查詢的時間范圍，但是效果并不好， 過了一會發現，這個服務fullGC是有問題的，太頻繁了，這個應該是導致介面超時的根本問題，因為時間也是對的上的， 這個是最近三個小時fullGC的監控圖：
這個是最近三天fullGC的監控圖：
對比一下，就不難發現，fullGC數量是從3月15號晚上9點開始增加的，也是這個介面對外開放的時間，
解決思路
1、首先去服務器上面下載dump檔案，分析是哪里造成了記憶體泄漏，頻繁觸發fullGC，首先找出服務器內java檔案的PID，然后保存dump檔案，我們公司java服務是固定埠號：1
使用top命令：
然后執行命令：jmap -dump:file=202303160924.dump,format=b 1 ，保存dump檔案
2、根據dump檔案，分析出堆內物件的分布情況
-
- 下載一個可以分析dump檔案的工具，這里我下載是Jprofiler
- 查看大物件的分析，發現是java.lang.ApplicationShutdownHooks的hooks占用太大記憶體，并且得知改熟悉是一個Map
- 分析這個Map里面的元素參考關系，可以看到這個map里面存的都是執行緒物件，并且大部分都是一個名為java.util.concurrent.ScheduledThreadPoolExecutor@59648a61的執行緒池物件，到了這里就定位到問題代碼了，是這次新加的介面里面有一個異步操作，用的guava并發包里面的一個超時等待功能的介面，具體思路就是啟用一個定時任務執行緒池去定時去檢查在規定時間內，是否回傳結果，
3、看看我的業務代碼是哪里出現了問題
//異步執行某個查詢方法，并且在規定時間內回傳查詢結果
public <T> T asyncWithTimeout(ScheduledThreadPoolExecutor executor, Callable<T> callable,
long time, TimeUnit unit) {
try {
ListeningExecutorService listeningExecutorService = MoreExecutors.listeningDecorator(threadPoolExecutor);
ListenableFuture<T> future = listeningExecutorService.submit(callable);
//這里是創建一個定時任務執行緒，去定時檢查是否在規定時間內查詢完畢，應該就是這個去添加了鉤子函式，進去看看
ScheduledExecutorService scheduledExecutorService = MoreExecutors.getExitingScheduledExecutorService(executor);
return Futures.withTimeout(future, time, unit, scheduledExecutorService).get(time, unit);
} catch (InterruptedException | ExecutionException | TimeoutException e) {
log.warn("異步方法執行失敗，error:{}", e.getMessage());
}
return null;
}
//=======================guava并發包代碼=======================
@Beta
@GwtIncompatible // TODO
public static ScheduledExecutorService getExitingScheduledExecutorService(
ScheduledThreadPoolExecutor executor) {
//每次都去創建一個新的物件
return new Application().getExitingScheduledExecutorService(executor);
}
final ScheduledExecutorService getExitingScheduledExecutorService(
ScheduledThreadPoolExecutor executor) {
return getExitingScheduledExecutorService(executor, 120, TimeUnit.SECONDS);
}
final ScheduledExecutorService getExitingScheduledExecutorService(
ScheduledThreadPoolExecutor executor, long terminationTimeout, TimeUnit timeUnit) {
useDaemonThreadFactory(executor);
ScheduledExecutorService service = Executors.unconfigurableScheduledExecutorService(executor);
//添加建構式的地方，進去看看
addDelayedShutdownHook(executor, terminationTimeout, timeUnit);
return service;
}
final void addDelayedShutdownHook(
final ExecutorService service, final long terminationTimeout, final TimeUnit timeUnit) {
checkNotNull(service);
checkNotNull(timeUnit);
//繼續點進去
addShutdownHook(
MoreExecutors.newThread(
//執行緒名字對上了，就在物件參考的截圖里面出現過
"DelayedShutdownHook-for-" + service,
new Runnable() {
@Override
public void run() {
try {
// We'd like to log progress and failures that may arise in the
// following code, but unfortunately the behavior of logging
// is undefined in shutdown hooks.
// This is because the logging code installs a shutdown hook of its
// own. See Cleaner class inside {@link LogManager}.
service.shutdown();
service.awaitTermination(terminationTimeout, timeUnit);
} catch (InterruptedException ignored) {
// We're shutting down anyway, so just ignore.
}
}
}));
}
@VisibleForTesting
void addShutdownHook(Thread hook) {
Runtime.getRuntime().addShutdownHook(hook);
}
//=======================guava并發包代碼=======================
public void addShutdownHook(Thread hook) {
SecurityManager sm = System.getSecurityManager();
if (sm != null) {
sm.checkPermission(new RuntimePermission("shutdownHooks"));
}
//定位到問題了，就是這里添加的鉤子函式
ApplicationShutdownHooks.add(hook);
}
static synchronized void add(Thread hook) {
if(hooks == null)
throw new IllegalStateException("Shutdown in progress");
if (hook.isAlive())
throw new IllegalArgumentException("Hook already running");
if (hooks.containsKey(hook))
throw new IllegalArgumentException("Hook previously registered");
//存在到 hooks 這個map物件里面，就是這個大物件
hooks.put(hook, hook);
}問題解決
經過上面問題的排查，造成hooks大物件的原因找到了，就是每次呼叫介面的時候，都會往hooks里面put一個物件， 所以，解決辦法很簡單，就是不用每次都去生成一個ScheduledExecutorService物件，類初始化的時候創建一次就行了 改造后的代碼如下：private ListeningExecutorService listeningExecutorService;
private ScheduledExecutorService scheduledExecutorService;
public static AsyncUtils getInstance() {
return ThreadHolder.INSTANCE.getAsyncWithCallback();
}
@SuppressWarnings("UnstableApiUsage")
private AsyncUtils() {
listeningExecutorService = MoreExecutors.listeningDecorator(ThreadPoolConstant.THREAD_POOL_EXECUTOR);
scheduledExecutorService = MoreExecutors.getExitingScheduledExecutorService(ThreadPoolConstant.SCHEDULED_THREAD_POOL_EXECUTOR);
}
@SuppressWarnings("UnstableApiUsage")
public <T> T asyncWithTimeout(Callable<T> callable,
long time, TimeUnit unit) {
try {
ListenableFuture<T> future = listeningExecutorService.submit(callable);
return Futures.withTimeout(future, time, unit, scheduledExecutorService).get(time, unit);
} catch (InterruptedException | ExecutionException | TimeoutException e) {
log.warn("異步方法執行失敗，error:{}", e.getMessage());
}
return null;
}
private enum ThreadHolder {
/**
* 執行緒持有類 INSTANCE
*/
INSTANCE;
private final AsyncUtils asyncUtils;
ThreadHolder() {
asyncUtils = new AsyncUtils();
}
public AsyncUtils getAsyncWithCallback() {
return asyncUtils;
}
}轉載請註明出處，本文鏈接：https://www.uj5u.com/houduan/547076.html
標籤：其他