– Written by Juan Gabriel Covas 2022-2024

Lsyncd uses a filesystem event interface (inotify or fsevents) to watch for changes to local files and directories. Lsyncd collates these events for several seconds and then spawns one or more processes to synchronize the changes to a remote filesystem.

Motivation: I wanted to compile the shiny, latest version of lsyncd 2.3.1 (2.3.0) instead of the stock version 2.2.2 or 2.2.3 to get some fixes and be able to use delete excluded from rsync (--delete-excluded), among other recent fixes.

Basically you're screwed if Lua is 5.4, which is the stock for EL 9 now.

So we need a full Lua 5.3 compiled from scratch, *including* the .so file (shared object) which is not easy to get…

1. Compiling Lua 5.3.6:

curl https://www.lua.org/ftp/lua-5.3.6.tar.gz > lua-5.3.6.tar.gz
tar -zxf lua-5.3.6.tar.gz
cd lua-5.3.6

We need to patch two Makefile files as seen at this reply from @dcarrith in this Stack Overflow question.

In file Makefile: Move this before TO_LIB=…, note the 6 for 5.3.6

# Lua version and release.
V= 5.3
R= $V.6

Then

TO_LIB= liblua.a

Becomes

TO_LIB= liblua.a liblua.so.$(R)

Now

     cd src && $(MAKE) $@

Becomes

     cd src && $(MAKE) $@ V=$(V) R=$(R)

Under section install: dummy, add:

  if [ ! -L $(INSTALL_LIB)/liblua.so.$(V) ] ;then ln -s $(INSTALL_LIB)/liblua.so.$(R) $(INSTALL_LIB)/liblua.so.$(V); fi
  if [ ! -L $(INSTALL_LIB)/liblua.so ] ;then ln -s $(INSTALL_LIB)/liblua.so.$(R) $(INSTALL_LIB)/liblua.so; fi

In file src/Makefile: The following

CFLAGS= -O2 -Wall -Wextra -DLUA_COMPAT_5_2 $(SYSCFLAGS) $(MYCFLAGS)

Becomes (add -fPIC)

CFLAGS= -O2 -Wall -Wextra -DLUA_COMPAT_5_2 $(SYSCFLAGS) $(MYCFLAGS) -fPIC

Then

LUA_A=  liblua.a

Becomes (add LUA_SO= line)

LUA_A=  liblua.a
LUA_SO= liblua.so

Now

ALL_T= $(LUA_A) $(LUA_T) $(LUAC_T)

Becomes

ALL_T= $(LUA_A) $(LUA_T) $(LUAC_T) $(LUA_SO)

Add this new block after $(LUA_A) block:

$(LUA_SO): $(CORE_O) $(LIB_O)
    $(CC) -shared -ldl -Wl,-soname,$(LUA_SO).$(V) -o $@.$(R) $? -lm $(MYLDFLAGS)
    if [ ! -L $(LUA_SO).$(V) ] ;then ln -sf $(LUA_SO).$(R) $(LUA_SO).$(V); fi
    if [ ! -L $(LUA_SO) ] ;then ln -sf $(LUA_SO).$(R) $(LUA_SO); fi

You must check that tabs are tabs, otherwise you'll get missing separators errors.

Now we can try to compile

  cd lua-5.3.6
  make INSTALL_TOP=/opt/lua-5.3.6 linux test

This should show:

[...]
src/lua -v
Lua 5.3.6  Copyright (C) 1994-2020 Lua.org, PUC-Rio

Ok now let's try to install:

  sudo make INSTALL_TOP=/opt/lua-5.3.6 linux install

We should have our .so file at /opt/lua-5.3.6/lib/liblua.so.5.3 and other stuff correctly setup

2. Compile lsyncd 2.3.1:

wget https://github.com/lsyncd/lsyncd/archive/refs/tags/v2.3.1.tar.gz -O lsyncd-2.3.1.tar.gz
tar xvzf lsyncd-2.3.1.tar.gz
cd lsyncd-2.3.1
mkdir build
cd build

Here we're going to patch everything to use 5.3.6 instead of stock 5.4 or whatever is in Enterprise Linux 9 now…

cmake .. -DLUA_INCLUDE_DIR=/opt/lua-5.3.6/include -DLUA_LIBRARIES=/opt/lua-5.3.6/lib/liblua.so.5.3.6 -DLUA_EXECUTABLE=/opt/lua-5.3.6/bin/lua -DLUA_COMPILER=/opt/lua-5.3.6/bin/luac

You can try to prepend CFLAGS=“-march=native” CXXFLAGS=“-march=native”, before cmake ..

We should see something like this:

-- The C compiler identification is GNU 11.4.1
-- The CXX compiler identification is GNU 11.4.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done

This is a good sign:

Found Lua: /usr/lib64/liblua-5.3.so (found version "5.3.6")

-- Configuring done (0.3s)
-- Generating done (0.0s)
-- Build files have been written to: /root/lsyncd-2.3.1/build

Now we can do (inside build directory)

make

Should end saying:

[100%] Built target lsyncd

Now we can install:

sudo make install

And…

cp /opt/lua-5.3.6/lib/liblua.so.5.3.6 /usr/lib64/liblua.so.5.3

So we can test:

lsyncd --help

WooHoo!

Now you only need:

/etc/lsyncd.conf

----
-- User configuration file for lsyncd.
--
-- Simple example for default rsync, but executing moves through on the target.
--
-- For more examples, see /usr/share/doc/lsyncd*/examples/
--

settings {
        logfile         = "/var/log/lsyncd/lsyncd.log",
        statusFile      = "/var/log/lsyncd/lsyncd.stat",
        statusInterval  = 60,
        nodaemon        = false,
        insist          = true,
--      maxProcesses    = 8,        -- will not spawn more than these number of processes. This adds across all sync{}s
--                                  -- but when using default.rsyncssh must have maxProcesses set to 1
--      maxDelays       = 1000      -- when this amount of delayed events is queued, actions will be spawned, even below the delay timer.
}

_my_dst_host = "root@somehost.domain.tld"

-- YOU NEED TO DEFINE SOMETHING TO WATCH FOR THE DAEMON TO SUCCESFULLY START: (uncomment some line, at must be valid)

_my_sites = {
--  { host=_my_dst_host,    source="/srv/www/somevhost/upload/",          targetdir="/srv/www-mirror-host/somedir/",                   delay=5 },
--  { host=_my_dst_host,    source="",      targetdir="", delay=10 },
}

for _idx, _site in pairs(_my_sites) do

    sync {
        default.rsyncssh,
        host        = _site['host'],
        source      = _site['source'],
        targetdir   = _site['targetdir'],
        delay       = _site['delay'],       -- The default rsync configuration will aggregate events up to delay seconds or 1000 separate uncollapsible events, which ever happens 1st
--      init        = false,                -- only valid if false, otherwise lsyncd fails to start
        delete      = true,
--                      thumbnails                 tmp                  dot files                                                office tmp
        exclude     = {'/**/th', '/*/tmp_import', '/GRUPO??/**/*.tmp', '/GRUPO??/**/.*', '/*/tmp_cron_mail', '/_adjuntos/temp', '~$*', '/c3perito'},
--      exclude     = {'/**/th', '/*/tmp_import', '/GRUPO??/**/jpegoptim-*.tmp'},
        rsync = {
            compress        = false,
            archive         = true,
            chown           = "lighttpd:lighttpd",
            delete_excluded = true,         -- delete excluded files on target if found
--          binary          = "/usr/local/bin/rsync"
            binary          = "/root/lsyncd-handlers/handler-lsyncd-rsync.sh"
        },
        ssh = {
            binary          = "/usr/bin/ssh",
--          binary          = "/usr/bin/handler-lsyncd-ssh.sh",
            identityFile    = "/root/.ssh/id_rsa_peritoline1",
            port            = 9876,
            _extra          = {"-o StrictHostKeyChecking=no"}
        }
    }

end

/etc/logrotate.d/lsyncd

/var/log/lsyncd/*.log /var/log/lsyncd/*.stat /var/log/lsyncd/*.txt {
    missingok
    notifempty
    daily
    rotate 7
    compress
    sharedscripts
}

/usr/lib/systemd/system/lsyncd.service

[root@cpline lsyncd-2.3.1]# cat /usr/lib/systemd/system/lsyncd.service
[Unit]
Description=Live Syncing (Mirror) Daemon
After=network.target

[Service]
Type=simple
EnvironmentFile=-/etc/sysconfig/lsyncd
ExecStart=/usr/bin/lsyncd -nodaemon $LSYNCD_OPTIONS

[Install]
WantedBy=multi-user.target

/etc/sysconfig/lsyncd

LSYNCD_OPTIONS="/etc/lsyncd.conf"

Please note that the following instructions are tested under CentOS 7 (soon to be EOL'ed)

1. Install devtoolset-11 modern dev tools

yum install centos-release-scl
yum install devtoolset-11

If you want a bash session for devtoolset-11, use:

# scl enable devtoolset-11 bash

If you want use devtoolset-11 inside a bash script, use:

source scl_source enable devtoolset-11

Check gcc version:

# gcc --version
gcc (GCC) 11.2.1 20220127 (Red Hat 11.2.1-9)

2. Ensure we have cmake and cmake3

yum install cmake cmake3

2.1 Now the problem is that cmake is v2.8 and cmake3 the version we need (Error message: CMake 3.10 or higher is required. You are running version 2.8.12.2). Thanks to this guide about Installing the latest git/cmake versions on RHEL/Centos.

The relevant part is to execute the following two commands, assuming you have installed cmake AND cmake3:

sudo alternatives --install /usr/local/bin/cmake cmake /usr/bin/cmake 10 \
--slave /usr/local/bin/ctest ctest /usr/bin/ctest \
--slave /usr/local/bin/cpack cpack /usr/bin/cpack \
--slave /usr/local/bin/ccmake ccmake /usr/bin/ccmake \
--family cmake

sudo alternatives --install /usr/local/bin/cmake cmake /usr/bin/cmake3 20 \
--slave /usr/local/bin/ctest ctest /usr/bin/ctest3 \
--slave /usr/local/bin/cpack cpack /usr/bin/cpack3 \
--slave /usr/local/bin/ccmake ccmake /usr/bin/ccmake3 \
--family cmake

Then we can set cmake3 as the default “cmake” command:

# alternatives --config cmake

There are 2 programs which provide 'cmake'.

Selection    Command
-----------------------------------------------
 + 1           cmake (/usr/bin/cmake)
*  2           cmake (/usr/bin/cmake3)

Enter to keep the current selection[+], or type selection number: 2

Test cmake version:

# cmake --version
cmake3 version 3.17.5

3. Install Lua 5.3

Lsyncd requires Lua version >= 5.2 so we're going to try Lua 5.3 from IUS repo for Centos 7

# yum install -y epel-release https://repo.ius.io/ius-release-el7.rpm

Make sure IUS repo is enabled:

# yum-config-manager --enable ius

Install all lua53u packages:

# yum install -y lua53u*

Please note that we want to install ALL packages, therefore lua53u* ←- with a trailing asterisk

Then later we can use the following at cmake .. line: -DLUA_INCLUDE_DIR=/usr/include/lua-5.3 -DLUA_LIBRARIES=/usr/lib64/liblua-5.3.so -DLUA_EXECUTABLE=/usr/bin/lua-5.3 -DLUA_COMPILER=/usr/bin/luac-5.3

Comment: althought the lsyncd manual says that LUA 5.2 at least is required, the stock version is Lua is 5.1 on CentOS 7. Build of lsyncd (cmake) seems to find OK the 5.1 LUA… so it's strange. For my simple /etc/lscynd.conf, seems to work, but finally got the way to use at least LUA 5.3 as described above.

4. Download latest version of lsyncd from github:

The 2.3.1 version supports “delete_excluded=true” at rsync portion of a sync config, which was important to me.

# wget https://github.com/lsyncd/lsyncd/archive/refs/tags/v2.3.1.tar.gz -O lsyncd-2.3.1.tar.gz
# tar xvzf lsyncd-2.3.1.tar.gz
# cd lsyncd-2.3.1

5. Build lsyncd

# mkdir build && cd build

If you want to try the stock LUA… which in CentOS 7 is 5.1 (not recommended)

# cmake ..

OR if you installed LUA 5.3 and want to force it, do the following:

Add this line to CMakeLists.txt file, above the line # building and compiling the part of lsyncd written in Lua

message(">>> Using LUA v" ${LUA_VERSION_STRING} " LUA_INCLUDE_DIR:" ${LUA_INCLUDE_DIR} " LUA_LIBRARIES:" ${LUA_LIBRARIES} " LUA_EXECUTABLE:" ${LUA_EXECUTABLE} " LUA_COMPILER:" ${LUA_COMPILER})

Run cmake this way:

# cmake .. -DLUA_INCLUDE_DIR=/usr/include/lua-5.3 -DLUA_LIBRARIES=/usr/lib64/liblua-5.3.so -DLUA_EXECUTABLE=/usr/bin/lua-5.3 -DLUA_COMPILER=/usr/bin/luac-5.3

You should see:

[...]
-- Found Lua: /usr/lib64/liblua-5.3.so (found version "5.3.4")
>>> Using LUA v5.3.4 LUA_INCLUDE_DIR:/usr/include/lua-5.3 LUA_LIBRARIES:/usr/lib64/liblua-5.3.so LUA_EXECUTABLE:/usr/bin/lua-5.3 LUA_COMPILER:/usr/bin/luac-5.3
[...]

If everything is ok we can try:

# make

And then:

# make install

This should end installing the following binary: /usr/local/bin/lsyncd.

Now for the good Lord I'm lazy to research how to create the systemctl lsyncd.service so being lsyncd a simple binary I just want to replace it with the proper, newer version.

So I want to install the RPM of stock lsyncd:

# yumdownloader lsyncd

I install the rpm with –no-deps because sometimes I compile rsync and the stock rsync RPM is a dependency of lsyncd… so I don't want it installed over the newer version.

# rpm -ivh --nodeps lsyncd-2.2.2-1.el7.x86_64.rpm

This will install the service and default config.

# lsyncd --version
Version: 2.2.2

Now we can just backup the unwanted lsyncd binary at /usr/bin:

# cp /usr/bin/lsyncd{,.bak} 

And move the 2.3.1 binary from /usr/local/bin into the bad, old binary:

# mv /usr/local/bin/lsyncd /usr/bin/lsyncd

Check version:

# lsyncd --version
Version: 2.3.1

And now we can edit /etc/lsyncd.conf, then

Use service lsyncd start and service lsyncd status to check and setup everything.

Logs are at /var/log/lsyncd

You're welcome.

Since lsyncd config files are LUA scripts, you can do something like this in a loop, configuring _my_sites for example:

Example conf file:

settings {
        logfile         = "/var/log/lsyncd/lsyncd.log",
        statusFile      = "/var/log/lsyncd/lsyncd.stat",
        statusInterval  = 60,
        nodaemon        = false,
        insist          = true
--      maxProcesses    = 8,        -- will not spawn more than these number of processes. This adds across all sync{}s
--                                  -- but when using default.rsyncssh must have maxProcesses set to 1
--      maxDelays       = 1000      -- when this amount of delayed events is queued, actions will be spawned, even below the delay timer.
}

_my_dst_host = "user@host"

_my_sites = {
    { host=_my_dst_host, source="/path/to/sync/",  targetdir="/path/to/target/",  delay=5 },
    { host=_my_dst_host, source="/path2/to/sync/", targetdir="/path2/to/target/", delay=5 }
}

for _idx, _site in pairs(_my_sites) do

    sync {
        default.rsyncssh,
        host        = _site['host'],
        source      = _site['source'],
        targetdir   = _site['targetdir'],
        delay       = 5,    -- default will aggregate events up to delay seconds or 1000 separate uncollapsible events, which ever happens first
        delete      = true,
--      exclude     = {'/**/th'}, -- exclude thumbnail dirs.
        rsync = {
            compress        = false,
            archive         = true,
--          chown           = "user:group",
            delete_excluded = true
        },
        ssh = {
            identityFile    = "/path/to/.ssh/id_rsa_host",
            port            = 22,
            _extra          = {"-o StrictHostKeyChecking=no"}
        }
    }

end

You can check other neat tricks here at the FAQ of lsyncd manual:

• https://lsyncd.github.io/lsyncd/faq/