Page 1 of 1

[ALPHA] AirDC++-WebAPI for QNAP NAS

PostPosted: 09 Dec 2016, 20:40
by NiQ
I've looked for a DC++ client for my NAS for a long time and didn't find any, so after around a month and a lot of headache I've managed to compile AirDC++-WebAPI for QNAP NAS products.

Before anything else, note that by proceeding you agree to the following disclaimer:

YOUR WARRANTY IS NOW VOID

This is not production grade software, it's hardly even beta - it's more like a "works for me" kind of thing. I am not a QNAP employee; I do not get paid for this. This is a community app and as such any warranties are disclaimed. You have hereby been warned that anything can happen and agree not to hold me responsible for anything which may occur, even if you end-up completely bricking your NAS or even if it explodes.

With that said, it is very unlikely that you'll actually end-up with a brick, however if you fail to follow these steps exactly you may end-up having to factory-reset your NAS, so backup anything important before proceeding!
This has been compiled and tested on a TS-653 Pro NAS. It should work for other QNAP machines running QTS4 as well but this is not guaranteed.
This is version 1.4.1, which is the latest version at the time of writing this post - however I do not promise to keep-up with each version.

CHANGELOG:
2016.12.29:
* Upgraded binaries to 1.4.1
* Tarball contains the dcppboot.xml file so no need to use -c anymore
* Instructions modified as needed

2016.12.09:
* Initial release, version 1.4.0

PREPARATION:

1. Install Entware-NG - Unfortunately this app cannot be released as a native QPKG. It's already pretty big and has too many dependencies on Entware-NG libraries.
Note: You must install the x86 (32-bit) version of Entware-NG. The x86 version can be installed even if your NAS has an amd64 (x86_64; 64-bit) CPU, but the 32-bit and 64-bit flavors of Entware-NG cannot coexist, i.e. if you already have the 64-bit version installed you must remove it before installing the 32-bit version. If you have other apps that depend on the 64-bit version then unfortunately you'll have to choose between them at this point.

2. Use OPKG to install the following packages: libpthread, libopenssl, boost-system, boost-regex, boost-thread, boost-chrono, boost-date_time, boost-atomic, libbz2, zlib, libminiupnpc, libiconv-full, libatomic, librt.

DOWNLOAD:

--> Download Link <--

Version archive:
1.4.0

INSTALLATION:

1. Extract the content of the archive into the root directory (the tarball already has the necessary directory prefixes). The executable will be placed in /opt/bin, libraries will be written to /opt/lib (This will also install libleveldb and libGeoIP, which are not yet available through OPKG) and some assets will be placed in /opt/share.

2. We will need to perform the initial configuration of AirDC++. By default it saves its configuration to ~/.airdcpp, but on the QNAP this is a tmpfs which is not large enough to store the configuration (which includes all shared file hashes) and also does not survive restarts, so we will need to make sure the configuration is stored on the hard drives -
UPDATE: As of version 1.4.1 using -c is no longer necessary, AirDC++ is already configured to use the correct directory, just run /opt/bin/airdcppd --configure.
The following two lines are for legacy purposes only:
Create the following directory: /opt/etc/airdcpp
Start the initial configuration: /opt/bin/airdcppd -c=/opt/etc/airdcpp --configure
IMPORTANT: You will need to set-up username and password for the web UI at this point, note that the password is stored plain text! Therefore do not use a password that is identical to your mail, bank account etc, and do not store the configuration in a directory that is shared by the NAS.

3. Start AirDC++: /opt/bin/airdcppd -d (Old version 1.4.0: You need to specify -c here as well).
(If you do not want the server to daemonize, remove the -d).

4. Fire up your favorite browser and browse to http://nasaddress:5600/. Log in with the user and password you've just set.

5. Click the down arrow on the top-right, choose settings. Select profile --> Advanced --> Miscellaneous. You will see a box named "NMDC encoding". Change the value to "ascii". This is important because apparently Entware's libiconv is buggy. If you do not set it or set it to anything else you will not be able to connect to most hubs.

6. Change any other settings as needed and connect.

KNOWN ISSUES:

1. i18n support - As noted, we have to set the encoding to "ascii" due to a buggy libiconv implementation, which means that files whose name contains non-latin characters will not display correctly. I currently have no solution for this and can't tell when or whether it'll be fixed (however if you happen to be an Entware-NG developer, you can assist!). This does not affect ADC hubs, which always use UTF-8.

2. Architecture - As noted, currently working on Entware-NG 32-bit only, which can be installed on 64-bit NAS but cannot coexist with the 64-bit of Entware-NG. ARM is completely incompatible. Compiling for Entware-NG 64-bit and/or Entware-NG ARM is probably possible but it's too much of a hassle for me. I'll post compiling instructions later on and you can try it yourself.

3. Packaging - Eventually I'll probably pack this into a decent OPKG to make installation easier (Note: OPKG, not QPKG - Entware-NG will always be required).

NOTES AND THANKS:

1. AirDC++-WebAPI is © 2011-2016 AirDC++ project, distributed under the GNU GPL license version 2 or any later version at your option.
The source code can be cloned from the project's git repository. No code has been modified (except for a two-character edit in a dependency) but there are a lot of weird compiler parameters needed to get this to compile. I'll post compiling instructions soon (if for any reason I don't please message me).

2. Many thanks to maksis from the AirDC++ development team, who's helped me a lot in debugging a lot of issues and a few weeks of correspondence with me in order to bring AirDC++ to the QNAP. It wouldn't have happened without him.

Re: [ALPHA] AirDC++-WebAPI for QNAP NAS

PostPosted: 13 Dec 2016, 20:13
by NiQ
COMPILING INSTRUCTIONS

You do not need to read this if you just want to use AirDC++ on your NAS. Just see the first message and download the binary. This is intended only should you wish to compile AirDC++ from source (e.g. new version, different CPU architecture). This is also intended for complete GPL adherence.
IMPORTANT: These instructions assume that you are familiar with the GNU/Linux shell and at least the standard methods of compiling packages from sources. If you do not I strongly suggest that you do not attempt to compile this yourself.
IN ANY CASE I WILL NOT BE HELD LIABLE FOR ANY DAMAGE RESULTING FROM ATTEMPTING ANYTHING WRITTEN HERE (SEE DISCLAIMER IN THE FIRST MESSAGE).

EDIT: Apparently the forum does not allow attaching files, so get the files needed for steps 9 and 13 from this link)

1. Install Entware-NG (according to your CPU architecture x86/amd64/arm).

2. Use opkg to install all necessary dependencies (listed in step 2 on the first message in this thread).

3. Consider also installing git through opkg, which will allow you to clone the sources directly to your NAS.

4. Download Node.js. Unfortunately it is not available from the opkg repository but the official download page has static Linux binaries so you don't have to compile it from source.

5. Install Node.js to /opt (just copy all folders in the tarball) so you can invoke npm directly.

6. Set-up your NAS for native compilation (see instructions). Make sure to download the include headers tarball and extract it to /opt/include and invoke the gcc_env.sh script (please read that page!).

7. Since GeoIP and LevelDB are not available through opkg you will have to compile them from source. Clone them from github.
If you've followed these instructions so far you should be able to compile GeoIP using the standard configure, make, make install procedure, but make sure to configure the install prefix to /opt (only the folders in /opt are part of Entware-NG, stored on the hard-drive and survive reboots!). Then copy the headers to /opt/include.
LevelDB does not have a configure and install script, so just build it with make and copy the binary to /opt/lib and the headers to /opt/include.

8. Clone websocketpp. Since this library is completely inline there's no need to compile anything, just copy it to /opt/include.

9. Apply the attached patch file to /opt/include/websocketpp/transport/asio/endpoint.hpp to disable IPv6 support (since QTS does not have an IPv6 stack any attempt to use the IPv6-enabled library will cause an immediate crash).

10. Download CMake. Again there are static Linux binaries so no need to compile from source (at least not for x86/amd64, if you're compiling for arm, sorry!)

11. Install CMake to /opt (again just extract the tarball to /opt).

12. Clone AirDC++ from the official git.

13. Download the attached Findwebsocketpp.cmake file and place it in the cmake subdirectory of the AirDC++ sources (this is needed because the version of CMake we're using does not have a native find script for websocketpp).

14. Invoke cmake using the following command (yes, it's long, for some reason CMake is unable to automatically locate most of the libraries on this platform):
cmake -DCMAKE_PREFIX_PATH=/opt -DCMAKE_INSTALL_PREFIX=/opt/ -DBZIP2_LIBRARIES=/opt/lib/libbz2.so.1.0 -DMINIUPNP_LIBRARY=/opt/lib/libminiupnpc.so.10 -DBoost_INCLUDE_DIR=/opt/include/ -DBoost_LIBRARIES=/opt/lib/ -DOPENSSL_ROOT_DIR=/opt -DOPENSSL_SSL_LIBRARY=/opt/lib/libssl.so.1.0.0 -DOPENSSL_CRYPTO_LIBRARY=/opt/lib/libcrypto.so.1.0.0 -DLibGeoIP_ROOT_DIR=/opt -DLibGeoIP_LIBRARY=/opt/lib/libGeoIP.so -DLibGeoIP_INCLUDE_DIR=/opt/include/GeoIP .

15. Watch the CMake output. At some point it will print the include directories and libraries to use. Make sure all of them are located in /opt (therefore part of Entware-NG). Do not use native libraries (outside /opt). Combining Entware-NG and native libraries in a binary will almost definitely result in a non-working binary if it even compiles (also, some of the native libraries are really broken). Do not proceed until only Entware-NG libraries are used.

16. Compile AirDC++ using make -j2

17. It will crash at around 98% (when linking the final executable) with unresolved externals to the atomic API. This is because for some reason the libboost-atomic library in Entware-NG does not include the necessary dependency on libatomic.
Unfortunately I haven't found a way of instructing CMake to include libatomic automatically so we have to compile it and let it crash once, then manually edit the makefile...
Edit the following files:
airdcpp-core/CMakeFiles/airdcpp.dir/link.txt
airdcpp-webapi/CMakeFiles/airdcpp-webapi.dir/link.txt
airdcppd/CMakeFiles/airdcppd.dir/link.txt
Make the same change in all of them - Look for -lboost_atomic and add -latomic immediately after it (do not delete anything).

18. Compile again using make -j2. This will be quicker because it will not actually recompile anything, just re-link. This time it will succeed.

19. Install using make install.

20. Go back to the first message in this thread and configure as usual.

Re: [ALPHA] AirDC++-WebAPI for QNAP NAS

PostPosted: 14 Dec 2016, 15:58
by maksis
Yeah, that's a welcome package. Some things that might need clarification:

NiQ wrote:CMake we're using does not have a native find script for websocketpp


Those files are not shipped with CMake, they are installed as part of the websocketpp package (https://airdcpp-web.github.io/docs/inst ... ketpp.html)

NiQ wrote:9. Apply the attached patch file to /opt/include/websocketpp/transport/asio/endpoint.hpp to disable IPv6 support (since QTS does not have an IPv6 stack any attempt to use the IPv6-enabled library will cause an immediate crash).


The application shouldn't attempt to use IPv6 if it's not supported by the operating system (fixed in version 1.4.0, https://github.com/airdcpp-web/airdcpp- ... issues/101)

NiQ wrote:It will crash at around 98% (when linking the final executable) with unresolved externals to the atomic API.


I'd really like to see the exact message that you receive. I've added a cmake file for libatomic already but I don't know how to test it myself.

NiQ wrote:By default it saves its configuration to ~/.airdcpp, but on the QNAP this is a tmpfs which is not large enough to store the configuration (which includes all shared file hashes) and also does not survive restarts, so we will need to make sure the configuration is stored on the hard drives


In that case, support for (CMAKE_INSTALL_PREFIX)/etc/airdcpp/ should possibly be added in the application itself to avoid the need for extra startup parameters. Update: https://airdcpp-web.github.io/docs/adva ... files.html

Generally speaking, the goal is to make the installation scripts and the application to work without OS-specific hacks whenever it makes sense.

Re: [ALPHA] AirDC++-WebAPI for QNAP NAS

PostPosted: 29 Dec 2016, 20:54
by NiQ
Thanks, I've updated to version 1.4.1 -

maksis wrote:Those files are not shipped with CMake, they are installed as part of the websocketpp package (https://airdcpp-web.github.io/docs/inst ... ketpp.html)

I've double checked and can't find that file in the websocketpp package. Presumably most distros will install that file along websocketpp and/or cmake but Entware-NG does not include either one at all and it doesn't seem that the Findwebsocket file is included in neither the original cmake nor the original websocketpp archives.

maksis wrote:The application shouldn't attempt to use IPv6 if it's not supported by the operating system (fixed in version 1.4.0, https://github.com/airdcpp-web/airdcpp- ... issues/101)

Good to know. Personally I kept the patch just in case some other software has that bug. I still wonder how come such a popular NAS OS doesn't have an IPv6 stack...

maksis wrote:I'd really like to see the exact message that you receive. I've added a cmake file for libatomic already but I don't know how to test it myself.

OK, here it is:
Code: Select all
[ 99%] Building CXX object airdcppd/CMakeFiles/airdcppd.dir/stacktrace.cpp.o
[100%] Linking CXX executable airdcppd
../airdcpp-core/libairdcpp.so.1.4.1: undefined reference to `__atomic_fetch_sub_8'
../airdcpp-core/libairdcpp.so.1.4.1: undefined reference to `__atomic_load_8'
../airdcpp-core/libairdcpp.so.1.4.1: undefined reference to `__atomic_store_8'
../airdcpp-core/libairdcpp.so.1.4.1: undefined reference to `__atomic_fetch_add_8'
collect2: error: ld returned 1 exit status
make[2]: *** [airdcppd/CMakeFiles/airdcppd.dir/build.make:215: airdcppd/airdcppd] Error 1
make[1]: *** [CMakeFiles/Makefile2:229: airdcppd/CMakeFiles/airdcppd.dir/all] Error 2
make: *** [Makefile:128: all] Error 2
[/share/Download/QNAP/airdcpp-webclient] #

I do think I have an explanation for that - Apparently libboost-atomic calls these functions, which are exported by libatomic. In most distros if you ldd libboost-atomic.so you'll find that it actually depends on libatomic.so, but for some reason in the Entware-NG distribution it's missing, which means that any binary that links with libboost-atomic will get that linker error - unless the binary also links with libatomic directly (which is what that link.txt patch does).

maksis wrote:In that case, support for (CMAKE_INSTALL_PREFIX)/etc/airdcpp/ should possibly be added in the application itself to avoid the need for extra startup parameters. Update: https://airdcpp-web.github.io/docs/adva ... files.html

Thanks! Already used this for the latest build.

maksis wrote:Generally speaking, the goal is to make the installation scripts and the application to work without OS-specific hacks whenever it makes sense.

I'll assist as much as needed.

Re: [ALPHA] AirDC++-WebAPI for QNAP NAS

PostPosted: 29 Dec 2016, 21:41
by maksis
NiQ wrote:I've double checked and can't find that file in the websocketpp package. Presumably most distros will install that file along websocketpp and/or cmake but Entware-NG does not include either one at all and it doesn't seem that the Findwebsocket file is included in neither the original cmake nor the original websocketpp archives.


It seems to be using different naming: https://github.com/zaphoyd/websocketpp/ ... #L253-L259

NiQ wrote:OK, here it is:
Code: Select all
[ 99%] Building CXX object airdcppd/CMakeFiles/airdcppd.dir/stacktrace.cpp.o
[100%] Linking CXX executable airdcppd
../airdcpp-core/libairdcpp.so.1.4.1: undefined reference to `__atomic_fetch_sub_8'
../airdcpp-core/libairdcpp.so.1.4.1: undefined reference to `__atomic_load_8'
../airdcpp-core/libairdcpp.so.1.4.1: undefined reference to `__atomic_store_8'
../airdcpp-core/libairdcpp.so.1.4.1: undefined reference to `__atomic_fetch_add_8'
collect2: error: ld returned 1 exit status
make[2]: *** [airdcppd/CMakeFiles/airdcppd.dir/build.make:215: airdcppd/airdcppd] Error 1
make[1]: *** [CMakeFiles/Makefile2:229: airdcppd/CMakeFiles/airdcppd.dir/all] Error 2
make: *** [Makefile:128: all] Error 2
[/share/Download/QNAP/airdcpp-webclient] #

I do think I have an explanation for that - Apparently libboost-atomic calls these functions, which are exported by libatomic. In most distros if you ldd libboost-atomic.so you'll find that it actually depends on libatomic.so, but for some reason in the Entware-NG distribution it's missing, which means that any binary that links with libboost-atomic will get that linker error - unless the binary also links with libatomic directly (which is what that link.txt patch does).


In fact, std::atomic is used in the application code directly. Are you able to get it working with help of https://github.com/airdcpp-web/airdcpp- ... omic.cmake ? I wonder if CMAKE_REQUIRED_LIBRARIES should be added for linking in https://github.com/airdcpp-web/airdcpp- ... eLists.txt as well.

Re: [ALPHA] AirDC++-WebAPI for QNAP NAS

PostPosted: 29 Dec 2016, 21:56
by NiQ
maksis wrote:It seems to be using different naming: https://github.com/zaphoyd/websocketpp/ ... #L253-L259

Hmm... I'm not even sure I've used cmake for websocketpp... just copied it to /opt/include because the whole library is inline. Maybe it'll work but I don't want to screw things up after I've already got them working.

maksis wrote:In fact, std::atomic is used in the application code directly. Are you able to get it working with help of https://github.com/airdcpp-web/airdcpp- ... omic.cmake ? I wonder if CMAKE_REQUIRED_LIBRARIES should be added for linking in https://github.com/airdcpp-web/airdcpp- ... eLists.txt as well.

I think only the latter is required. Even without the .cmake file, the linker already knows where to find libatomic.so, it just doesn't know it needs to link with it.
Adding it as a required library should do the trick but you should make sure it doesn't break any other platform. Maybe it should be configurable.