9
My team implemented a Pipeline using Computer Vision (Opencv) and a DNN (Neural Network) in Tensorflow and Keras using C++. This Pipeline is an AWS "Layer" used by a Lambda Function implemented in Python and this "Layer" is invoked using the Boost library.
A container has also been created in Docker (amazonlinux) with all requirements (Opencv, Boost, Python 3.7, serverless, etc.) to build the Pipeline code (using cmake). Everything was working fine, the Pipeline and Lambda Functions build was running smoothly.
Well, let’s go into the detail of my question: to improve the performance of Pipeline, I want to measure the duration/time of several stages of this Pipeline. So I decided to do this using aws-sdk-cpp-Xray, because AWS X-Ray is a great resource to achieve my goal.
Before starting coding, I added and built aws-sdk-cpp on my local machine (Macos) and in the same container (amazonlinux) of the Docker (via dockerfile / Docker-Compose) that I use to build the Lambda Functions pipeline (below is the part I added in Dockerfile):
# Build AWS SDK with BUILD_ONLY XRay
RUN mkdir /tmp/${AWS_SDK_CPP}/build
WORKDIR /tmp/${AWS_SDK_CPP}/build
RUN cmake -D CMAKE_BUILD_TYPE=RELEASE \
-D BUILD_ONLY="xray" \
-D TARGET_ARCH=LINUX \
-D CMAKE_INSTALL_PREFIX=/usr/local \
-D ENABLE_TESTING=OFF \
-D SIMPLE_INSTALL=OFF \
-D BUILD_SHARED_LIBS=ON \
-D BUILD_DEPS=ON \
..
RUN make -j $(nproc) && make install
Then I put the aws-sdk-cpp-Xray references in Cmakelist.txt of my project c++ (the pipeline) as below:
cmake_minimum_required(VERSION 3.4.1)
set( CMAKE_CXX_STANDARD 11 )
find_package(AWSSDK REQUIRED COMPONENTS xray)
add_definitions(-DUSE_IMPORT_EXPORT)
SET(GCC_COVERAGE_COMPILE_FLAGS "-rdynamic -O3 -ffunction-sections -fdata-sections")
add_definitions(${GCC_COVERAGE_COMPILE_FLAGS})
find_package(OpenCV REQUIRED)
set( LIB_FOLDER /lib/src )
link_libraries(lib ${OpenCV_LIBS} ${AWSSDK_LINK_LIBRARIES})
target_link_libraries(lib ${OpenCV_LIBS})
include_directories(${LIB_FOLDER}/include)
file(GLOB lib_src
"${LIB_FOLDER}/**/**.cpp"
)
include_directories(${CMAKE_INSTALL_PREFIX})
file(GLOB aws_sdk_src
"${CMAKE_INSTALL_PREFIX}/**/**/**.*"
)
add_library(lib SHARED ${lib_src} ${aws_sdk_src})
Finally, I tried to encode Xray using aws-sdk-cpp-Xray in c++ in various ways within the Pipeline project, but without much success (due to Amazon’s lack of c++ examples of Xray). And when I tried to test the execution of Lambda Function that invokes Layer from Pipeline with the new implementations of Xray I started getting the error below:
Runtime.ImportModuleError: Unable to import module 'functions/myfunction': /opt/lib/lib.so: undefined symbol:
_ZTVN3Aws35AmazonSerializableWebServiceRequestE
Where:
- "myfunction" is the Lambda Function in Python.
- "lib.so" is Layer (Pipeline) in c++.
Some hypotheses: It seems that the execution problem is caused by the installation of aws-sdk-cpp in Dockerfile. Or maybe it was caused by entries in the Pipeline Cmakelist.txt project file.. Or even missing some * files. so, * . a or . only to the Layer (Pipeline) package. But due to lack of documentation / manual and nothing found on the above errors, I am asking for help with some example on using AWS Xray using C++ or even alternatives to implement this time measurement.
Note: I also asked for examples on the AWS SDK Github home. Please, if possible, help me by voting on the example request: https://github.com/awsdocs/aws-doc-sdk-examples/issues/1012
Thank you!