在使用WebRTC的時候，對視頻進行美顏處理一般有兩種方式：替換WebRTC中的采集模塊和對視頻資料進行美顏，
一、替換WebRTC中的采集模塊
替換WebRTC中的采集模塊，相對比較簡單，使用GPUImageVideoCamera替換WebRTC中的視頻采集，得到經過GPUImage添加美顏處理后的影像，發送給WebRTC的OnFrame方法，
參考基于WebRTC框架開發的全平臺推拉流SDK：Github
設定美顏
- (void)setBeautyFace:(BOOL)beautyFace{
if(_beautyFace == beautyFace) return;
_beautyFace = beautyFace;
[_emptyFilter removeAllTargets];
[_filter removeAllTargets];
[_videoCamera removeAllTargets];
if(_beautyFace){
_filter = [[GPUImageBeautifyFilter alloc] init];
_emptyFilter = [[GPUImageEmptyFilter alloc] init];
}else{
_filter = [[GPUImageEmptyFilter alloc] init];
}
__weak typeof(self) _self = self;
[_filter setFrameProcessingCompletionBlock:^(GPUImageOutput *output, CMTime time) {
[_self processVideo:output];
}];
[_videoCamera addTarget:_filter];
if (beautyFace) {
[_filter addTarget:_emptyFilter];
if(_gpuImageView) [_emptyFilter addTarget:_gpuImageView];
} else {
if(_gpuImageView) [_filter addTarget:_gpuImageView];
}
}
格式轉換
GPUImage處理后的Pixel格式為BGRA,當處理完成后需要轉換為I420格式，用于內部處理和渲染，
WebRTC 在編碼的時候使用的是NV12格式的Pixel，所以在編碼的時候會進行二次格式轉換
-(void) processVideo:(GPUImageOutput *)output{
rtc::CritScope cs(&cs_capture_);
if (!_isRunning) {
return;
}
@autoreleasepool {
GPUImageFramebuffer *imageFramebuffer = output.framebufferForOutput;
size_t width = imageFramebuffer.size.width;
size_t height = imageFramebuffer.size.height;
uint32_t size = width * height * 3 / 2;
if(self.nWidth != width || self.nHeight != height)
{
self.nWidth = width;
self.nHeight = height;
if(_dst)
delete[] _dst;
_dst = NULL;
}
if(_dst == NULL)
{
_dst = new uint8_t[size];
}
uint8_t* y_pointer = (uint8_t*)_dst;
uint8_t* u_pointer = (uint8_t*)y_pointer + width*height;
uint8_t* v_pointer = (uint8_t*)u_pointer + width*height/4;
int y_pitch = width;
int u_pitch = (width + 1) >> 1;
int v_pitch = (width + 1) >> 1;
libyuv::ARGBToI420([imageFramebuffer byteBuffer], width * 4, y_pointer, y_pitch, u_pointer, u_pitch, v_pointer, v_pitch, width, height);
if(self.bVideoEnable)
libyuv::I420Rect(y_pointer, y_pitch, u_pointer, u_pitch, v_pointer, v_pitch, 0, 0, width, height, 32, 128, 128);
if(_capturer != nil)
_capturer->CaptureYUVData(_dst, width, height, size);
}
}
美顏后的資料發送給WebRTC的OnFrame方法
GPUImageVideoCapturer 類為GPUImage 封裝的攝像頭類，跟WebRTC中的采集類功能保持一致，繼承 cricket::VideoCapturer 類，便可以往WebRTC中塞入采集的音視頻流，
namespace webrtc {
// 繼承cricket::VideoCapturer
class GPUImageVideoCapturer : public cricket::VideoCapturer {
...
}
}
void GPUImageVideoCapturer::CaptureYUVData(const webrtc::VideoFrame& frame, int width, int height)
{
VideoCapturer::OnFrame(frame, width, height);
}
二、對視頻資料進行美顏
對視頻資料美顏的思路就是傳統的第三方美顏SDK的做法，對內部采集的音視頻資料進行處理：內部采集的資料（CVPixelBufferRef）-》轉換為紋理（GLuint）-》對紋理進行音視頻的美顏-》美顏的紋理轉換為iOS的采集資料（CVPixelBufferRef）-》回傳給WebRTC內部進行渲染編碼和傳輸，
同步執行緒
內部處理的一般都是使用同步執行緒，這樣能夠保證資料線性流動，參閱GPUImage中的代碼片段
runSynchronouslyOnVideoProcessingQueue(^{
// 美顏處理
});
把CVPixelBufferRef 資料轉換為紋理（GLuint）
RGB格式型別的轉換方式
-
CoreVideo框架的方法：使用此方法可以創建CVOpenGLESTextureRef紋理，并通過CVOpenGLESTextureGetName(texture)獲取紋理id，- (GLuint)convertRGBPixelBufferToTexture:(CVPixelBufferRef)pixelBuffer { if (!pixelBuffer) { return 0; } CGSize textureSize = CGSizeMake(CVPixelBufferGetWidth(pixelBuffer), CVPixelBufferGetHeight(pixelBuffer)); CVOpenGLESTextureRef texture = nil; CVReturn status = CVOpenGLESTextureCacheCreateTextureFromImage(nil, [[GPUImageContext sharedImageProcessingContext] coreVideoTextureCache], pixelBuffer, nil, GL_TEXTURE_2D, GL_RGBA, textureSize.width, textureSize.height, GL_BGRA, GL_UNSIGNED_BYTE, 0, &texture); if (status != kCVReturnSuccess) { NSLog(@"Can't create texture"); } self.renderTexture = texture; return CVOpenGLESTextureGetName(texture); } -
OpenGL的方法：創建紋理物件，使用glTexImage2D方法上傳CVPixelBufferRef中影像資料data到紋理物件中，glBindTexture(GL_TEXTURE_2D, [outputFramebuffer texture]); glTexImage2D(GL_TEXTURE_2D, 0, _pixelFormat==GPUPixelFormatRGB ? GL_RGB : GL_RGBA, (int)uploadedImageSize.width, (int)uploadedImageSize.height, 0, (GLint)_pixelFormat, (GLenum)_pixelType, bytesToUpload);
YUV格式型別的轉換方式
- (GLuint)convertYUVPixelBufferToTexture:(CVPixelBufferRef)pixelBuffer {
if (!pixelBuffer) {
return 0;
}
CGSize textureSize = CGSizeMake(CVPixelBufferGetWidth(pixelBuffer),
CVPixelBufferGetHeight(pixelBuffer));
[EAGLContext setCurrentContext:self.context];
GLuint frameBuffer;
GLuint textureID;
// FBO
glGenFramebuffers(1, &frameBuffer);
glBindFramebuffer(GL_FRAMEBUFFER, frameBuffer);
// texture
glGenTextures(1, &textureID);
glBindTexture(GL_TEXTURE_2D, textureID);
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, textureSize.width, textureSize.height, 0, GL_RGBA, GL_UNSIGNED_BYTE, NULL);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, textureID, 0);
glViewport(0, 0, textureSize.width, textureSize.height);
// program
glUseProgram(self.yuvConversionProgram);
// texture
CVOpenGLESTextureRef luminanceTextureRef = nil;
CVOpenGLESTextureRef chrominanceTextureRef = nil;
CVReturn status = CVOpenGLESTextureCacheCreateTextureFromImage(kCFAllocatorDefault,
self.textureCache,
pixelBuffer,
nil,
GL_TEXTURE_2D,
GL_LUMINANCE,
textureSize.width,
textureSize.height,
GL_LUMINANCE,
GL_UNSIGNED_BYTE,
0,
&luminanceTextureRef);
if (status != kCVReturnSuccess) {
NSLog(@"Can't create luminanceTexture");
}
status = CVOpenGLESTextureCacheCreateTextureFromImage(kCFAllocatorDefault,
self.textureCache,
pixelBuffer,
nil,
GL_TEXTURE_2D,
GL_LUMINANCE_ALPHA,
textureSize.width / 2,
textureSize.height / 2,
GL_LUMINANCE_ALPHA,
GL_UNSIGNED_BYTE,
1,
&chrominanceTextureRef);
if (status != kCVReturnSuccess) {
NSLog(@"Can't create chrominanceTexture");
}
glActiveTexture(GL_TEXTURE0);
glBindTexture(GL_TEXTURE_2D, CVOpenGLESTextureGetName(luminanceTextureRef));
glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
glUniform1i(glGetUniformLocation(self.yuvConversionProgram, "luminanceTexture"), 0);
glActiveTexture(GL_TEXTURE1);
glBindTexture(GL_TEXTURE_2D, CVOpenGLESTextureGetName(chrominanceTextureRef));
glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
glUniform1i(glGetUniformLocation(self.yuvConversionProgram, "chrominanceTexture"), 1);
GLfloat kXDXPreViewColorConversion601FullRange[] = {
1.0, 1.0, 1.0,
0.0, -0.343, 1.765,
1.4, -0.711, 0.0,
};
GLuint yuvConversionMatrixUniform = glGetUniformLocation(self.yuvConversionProgram, "colorConversionMatrix");
glUniformMatrix3fv(yuvConversionMatrixUniform, 1, GL_FALSE, kXDXPreViewColorConversion601FullRange);
// VBO
glBindBuffer(GL_ARRAY_BUFFER, self.VBO);
GLuint positionSlot = glGetAttribLocation(self.yuvConversionProgram, "position");
glEnableVertexAttribArray(positionSlot);
glVertexAttribPointer(positionSlot, 3, GL_FLOAT, GL_FALSE, 5 * sizeof(float), (void*)0);
GLuint textureSlot = glGetAttribLocation(self.yuvConversionProgram, "inputTextureCoordinate");
glEnableVertexAttribArray(textureSlot);
glVertexAttribPointer(textureSlot, 2, GL_FLOAT, GL_FALSE, 5 * sizeof(float), (void*)(3* sizeof(float)));
glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);
glDeleteFramebuffers(1, &frameBuffer);
glBindFramebuffer(GL_FRAMEBUFFER, 0);
glBindBuffer(GL_ARRAY_BUFFER, 0);
glFlush();
self.luminanceTexture = luminanceTextureRef;
self.chrominanceTexture = chrominanceTextureRef;
if (luminanceTextureRef) {
CFRelease(luminanceTextureRef);
}
if (chrominanceTextureRef) {
CFRelease(chrominanceTextureRef);
}
return textureID;
}
使用GPUImageTextureInput 加載濾鏡和使用GPUImageTextureOutput輸出資料
[GPUImageContext setActiveShaderProgram:nil];
GPUImageTextureInput *textureInput = [[ARGPUImageTextureInput alloc] initWithTexture:textureID size:size];
GPUImageSmoothToonFilter *filter = [[GPUImageSmoothToonFilter alloc] init];
[textureInput addTarget:filter];
GPUImageTextureOutput *textureOutput = [[GPUImageTextureOutput alloc] init];
[filter addTarget:textureOutput];
[textureInput processTextureWithFrameTime:kCMTimeZero];
得到textureOutput，即得到輸出的紋理，
GPUImageTextureOutput紋理轉化為CVPixelBufferRef 資料
- (CVPixelBufferRef)convertTextureToPixelBuffer:(GLuint)texture
textureSize:(CGSize)textureSize {
[EAGLContext setCurrentContext:self.context];
CVPixelBufferRef pixelBuffer = [self createPixelBufferWithSize:textureSize];
GLuint targetTextureID = [self convertRGBPixelBufferToTexture:pixelBuffer];
GLuint frameBuffer;
// FBO
glGenFramebuffers(1, &frameBuffer);
glBindFramebuffer(GL_FRAMEBUFFER, frameBuffer);
// texture
glBindTexture(GL_TEXTURE_2D, targetTextureID);
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, textureSize.width, textureSize.height, 0, GL_RGBA, GL_UNSIGNED_BYTE, NULL);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, targetTextureID, 0);
glViewport(0, 0, textureSize.width, textureSize.height);
// program
glUseProgram(self.normalProgram);
// texture
glActiveTexture(GL_TEXTURE0);
glBindTexture(GL_TEXTURE_2D, texture);
glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
glUniform1i(glGetUniformLocation(self.normalProgram, "renderTexture"), 0);
// VBO
glBindBuffer(GL_ARRAY_BUFFER, self.VBO);
GLuint positionSlot = glGetAttribLocation(self.normalProgram, "position");
glEnableVertexAttribArray(positionSlot);
glVertexAttribPointer(positionSlot, 3, GL_FLOAT, GL_FALSE, 5 * sizeof(float), (void*)0);
GLuint textureSlot = glGetAttribLocation(self.normalProgram, "inputTextureCoordinate");
glEnableVertexAttribArray(textureSlot);
glVertexAttribPointer(textureSlot, 2, GL_FLOAT, GL_FALSE, 5 * sizeof(float), (void*)(3* sizeof(float)));
glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);
glDeleteFramebuffers(1, &frameBuffer);
glBindFramebuffer(GL_FRAMEBUFFER, 0);
glBindBuffer(GL_ARRAY_BUFFER, 0);
glFlush();
return pixelBuffer;
}
把美顏后的CVPixelBufferRef同步回傳給SDK，進行渲染傳輸，
三、總結
對音視頻的美顏，已經成為了音視頻應用的常用功能，除了上述兩種做法外，還可以使用第三方美顏，一般音視頻廠商都有提供自采集功能，而第三方美顏功能則提供有采集美顏相機功能，二者正好可以無縫結合，如果自身的應用中對美顏要求不是很高，采用音視頻SDK自帶的美顏即可（美白、美顏、紅潤），如果用在娛樂場景，除了美顏，還要美型（廋臉，大眼）、貼紙（2D、3D）的，必須要集成第三方美顏SDK了，
轉載請註明出處，本文鏈接：https://www.uj5u.com/qita/296777.html
標籤：其他