Jump to content

KVS main server load balancing


leonrak

Recommended Posts

200k daily visitors is very small traffic for KVS. This should not result in any performance issue, even on middle-end server. Please contact support to see some performance evaluation.

Load balancing of the main server is only possible by using satellite, configured on a separate server under the same domain. However the main problem is that using satellite will increase database load twofold, which may be another performance issue. Load balancing of database is not yet possible, we are still working with that.

  • Like 1
Link to comment
Share on other sites

Hi,

I have -

Active videos 370,000

Active albums: 50,000

Members: 14282

Traffic 250K daily

Traffic: https://prnt.sc/163ffqw

 

And Main Server:

2x Intel Xeon Silver 4214 - 12c/ 24t - 2.2GHz/ 3.2GHz

Memory : 96GB

2x960 SSD

1Gbps

And DB Server:

Intel Xeon-D 2141I - 8c/ 16t - 2.2GHz/ 3GHz

Memory : 64GB

2x500 SSD,

1Gbps

 

But still getting errors and load much, Both Server LA: 5-10/ 

 

I need some suggestion guys.

 

and my thoughts are remove unused JavaScript & CSS and load  Page by Page with $smart If condition.

Like: If a user on Video-list, load only video-list and header-footer codes,

And move some grabbers from youtube-dl to only PHP, because youtube-dl is also taking CPU, & Memory. or Optimized tasks for parallel process, because Conversion is on Remote Server and Storage is also Remote Server, 

 

But I don't know much about $smarty, maybe after remove codes $smarty take place for the loads.

 

 

Edited by xvids
Link to comment
Share on other sites

12 hours ago, xvids said:

But still getting errors and load much, Both Server LA: 5-10/ 

The number of errors you have is OK for your traffic. Please note that you have only real users as 250k daily, but you most likely have 1m of overall traffic (including bots). The load is caused not by only users, but by bots as well (SEO bots, DMCA bots). The more videos you have, the more bots you have.

Average LA 5-10 is also OK.

Most probably errors happen at the time when your server has peak traffic + some grabbers start to operate. In order to get rid of this we should probably think to allow scheduling grabber to some specific times. We also noticed that you have Template Cache Cleanup plugin configured every 6 hours:

template_cache_cleanup.png.04cf557ce3644de9963e36bf39abbd0f.png

This is not a good idea, better configure it once in 24 hours and time of day = choose the time when you have less traffic. This plugin increases HDD load and may interfere with imports and traffic and produce higher server load.

Link to comment
Share on other sites

19 minutes ago, Tech Support said:

Average LA 5-10 is also OK

yes LA 5-10 is OK but some time its goes too up 50-60, some tasks wake ffmpeg on local server, can I disable local server ffmpeg?

 

22 minutes ago, Tech Support said:

This is not a good idea, better configure it once in 24 hours and time of day = choose the time when you have less traffic.

 

I have configured 24h, but I think this will not able to clean for millions file, that's why I configure to 6h. 

Link to comment
Share on other sites

29 minutes ago, Tech Support said:

The number of errors you have is OK for your traffic. Please note that you have only real users as 250k daily, but you most likely have 1m of overall traffic (including bots). The load is caused not by only users, but by bots as well (SEO bots, DMCA bots). The more videos you have, the more bots you have

Yes you are right as I see in CloudFlare in 24h. 10million request, https://prnt.sc/1662m0v

I want to block Bing bot, is this good Idea? If good, the please tell me how I can block Bing,

because many DMCA bot using Bing API to scan websites, as I see many DMCA agents send Invalid URLs,

Link to comment
Share on other sites

1 hour ago, xvids said:

yes LA 5-10 is OK but some time its goes too up 50-60, some tasks wake ffmpeg on local server, can I disable local server ffmpeg?

Local ffmpeg is used to validate the uploaded video files, but it is not used for conversion. So it should not take much resources.

 

1 hour ago, xvids said:

I have configured 24h, but I think this will not able to clean for millions file, that's why I configure to 6h. 

It is possible to reduce the number of files in cache by increasing memory caching and disabling file caching for some blocks. But in your case you have another issue - your memcache is configured with a very small memory limit (64mb), so it cannot be effectively used. Please ask host support to update memcache configuration to allow 2GB. You can check that in Administration -> Installation info:

memcache.png.86123b8418afc3658c93312b4a83f3b3.png

 

58 minutes ago, xvids said:

I want to block Bing bot, is this good Idea? If good, the please tell me how I can block Bing,

I'm not SEO expert, but I think blocking bots is not a good idea, as it may result in your site being removed from search.

Link to comment
Share on other sites

You can now reduce the number of cache files by setting cache to 0 for video_view and video_comments blocks:

memcache2.thumb.png.b86c67cf2bae5242f132acf7b925542d.png

 

But at the same time you should increase page cache to 86400 to increase the % of successful cache hits for the whole video page. This will greatly reduce the number of files in cache.

Link to comment
Share on other sites

13 hours ago, xvids said:

Memcache memory Increased to 2GB, Now??

I think 64 mb memcached memory very small, incrase minumum 2-4 Gb.

My tube using kvs on centos/apache fpm dialy unique 25k/ total 100k and MemCache load: 975 Mb / 4 Gb (23%)).  LA peak 0.8 without convert.

I think you have a problem with the apache / nginx / sql settings.

Or I saw in the screenshot that you are converting thousands of movies every day, then the load goes high on any fast server.

I saw you have a separate database server, I think you should transfer the conversion there not to use it locally.


 

 

Link to comment
Share on other sites

I have Ubuntu 20.04:

I edited some Nginx and Apache config. Now L.A 5-6 because All time background task in process,

But now Database on MariaDB is working Slow, Please suggest me some info for DB, I have seen many answer on Stackweb that convert MyIsam to Innodb,

Intel Xeon-D 2141I - 8c/ 16t - 2.2GHz/ 3GHz

Memory : 64GB

Ubuntu: 20.04

[client]
port=3306
socket=/var/run/mysqld/mysqld.sock

[mysqld_safe]
socket=/var/run/mysqld/mysqld.sock
open_files_limit=4294967295

[mysqld]
user=mysql
pid-file=/var/run/mysqld/mysqld.pid
socket=/var/run/mysqld/mysqld.sock
port=3306
basedir=/usr
datadir=/var/lib/mysql
tmpdir=/tmp
lc-messages-dir=/usr/share/mysql
log_error=/var/log/mysql/error.log

symbolic-links=0
local-infile=0

skip-name-resolve
skip-external-locking
key_buffer_size = 1024M
max_allowed_packet = 1G
thread_stack = 384K
table_open_cache = 512
sort_buffer_size = 11M
read_buffer_size = 21M
read_rnd_buffer_size = 24M
myisam_sort_buffer_size = 512M
myisam_block_size = 5120
myisam_data_pointer_size = 7

key_cache_block_size = 8192
key_cache_division_limit = 60
key_cache_file_hash_size = 8192
key_cache_segments = 0

thread_cache_size = 256
query_cache_size= 300M
query_cache_type = 1
query_cache_limit = 256K
query_cache_min_res_unit = 2k

tmp_table_size= 256M
max_heap_table_size= 256M

#innodb_use_native_aio = 0
innodb_file_per_table

max_connections=2000
max_user_connections=500
wait_timeout=10
interactive_timeout=50
long_query_time=5

!includedir /etc/mysql/conf.d/
!includedir /etc/mysql/mariadb.conf.d/

Apache config:

 

# Global configuration
PidFile ${APACHE_PID_FILE}
Timeout 40
KeepAlive On
MaxKeepAliveRequests 3000
KeepAliveTimeout 5

<IfModule mpm_prefork_module>
    ServerLimit         256
    StartServers        10
    MinSpareServers     75
    MaxSpareServers     256
    MaxRequestWorkers   4500
    MaxConnectionsPerChild 10000
</IfModule>

<IfModule mpm_worker_module>
    ServerLimit            256
    StartServers           10
    MinSpareThreads        75
    MaxSpareThreads        256
    ThreadLimit            64
    ThreadsPerChild        32
    MaxRequestWorkers      4500
    MaxConnectionsPerChild 10000
</IfModule>

<IfModule mpm_event_module>
    StartServers        10
    MinSpareThreads     75
    MaxSpareThreads     256
    ThreadLimit         64
    ThreadsPerChild     32
    MaxRequestWorkers   4500
    MaxConnectionsPerChild 10000
</IfModule>

 

 

Nginx config:

 

 

# Server globals
user                    www-data;
worker_processes        24;
worker_rlimit_nofile    65535;
error_log               /var/log/nginx/error.log;
pid                     /var/run/nginx.pid;

include /etc/nginx/modules-enabled/*.conf;

# Worker config
events {
        worker_connections  4096;
        use                 epoll;
        multi_accept        off;
        accept_mutex        off;
}

http {
    # Main settings
    sendfile                        on;
        aio                             threads;
        directio                        8m;
    tcp_nopush                      on;
    tcp_nodelay                     on;
    client_header_timeout           180s;
    client_body_timeout             180s;
    client_header_buffer_size       2k;
    client_body_buffer_size         256k;
    client_max_body_size            256m;
    large_client_header_buffers     4 8k;
    send_timeout                    60s;
    keepalive_timeout               60s;
    keepalive_requests              20000;
    reset_timedout_connection       on;
    server_tokens                   off;
    server_name_in_redirect         off;
    server_names_hash_max_size      512;
    server_names_hash_bucket_size   512;
    charset                         utf-8;

    # FastCGI settings
    fastcgi_buffers                 4 256k;
    fastcgi_buffer_size             256k;
    fastcgi_busy_buffers_size       256k;
    fastcgi_temp_file_write_size    256k;
    fastcgi_connect_timeout         30s;
    fastcgi_read_timeout            300s;
    fastcgi_send_timeout            180s;
    fastcgi_cache_lock              on;
    fastcgi_cache_lock_timeout      5s;
    fastcgi_cache_background_update on;
    fastcgi_cache_revalidate        on;

    # Proxy settings
    proxy_redirect                  off;
    proxy_set_header                Host $host;
    proxy_set_header                X-Real-IP $remote_addr;
    proxy_set_header                X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_http_version              1.1;
    proxy_pass_header               Set-Cookie;
    proxy_buffers                   32 4k;
    proxy_connect_timeout           30s;
    proxy_read_timeout              300s;
    proxy_send_timeout              180s;

    # Log format
    log_format  main    '$remote_addr - $remote_user [$time_local] $request '
                        '"$status" $body_bytes_sent "$http_referer" '
                        '"$http_user_agent" "$http_x_forwarded_for"';
    log_format  bytes   '$body_bytes_sent';
    log_not_found off;
    access_log off;

    # Mime settings
    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    # Compression
    gzip                on;
    gzip_static         on;
    gzip_vary           on;
    gzip_comp_level     6;
    gzip_min_length     1024;
    gzip_buffers        16 8k;
    gzip_http_version   1.1;
    gzip_types          text/plain text/css text/javascript text/js text/xml application/json application/javascript application/x-javascript application/xml application/xml+rss applicatio>
    gzip_proxied        any;
    gzip_disable        "MSIE [1-6]\.";

    # Cloudflare https://www.cloudflare.com/ips
    set_real_ip_from 103.21.244.0/22;
    # SSL PCI compliance
    ssl_session_cache   shared:SSL:20m;
    ssl_session_timeout 60m;
    ssl_buffer_size     1400;
    ssl_protocols       TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers on;
    ssl_ciphers         
    ssl_dhparam         /etc/ssl/dhparam.pem;
    ssl_ecdh_curve      secp384r1;
    ssl_session_tickets off;
    resolver 213.186.33.99  valid=300s ipv6=off;
    resolver_timeout    5s;

    # Error pages
    error_page 403 /error/404.html;
    error_page 404 /error/404.html;
    error_page 410 /error/410.html;
    error_page 500 501 502 503 504 505 /error/50x.html;

    # Proxy cache
    proxy_cache_path /var/cache/nginx levels=2 keys_zone=cache:10m inactive=60m max_size=1024m;
    proxy_cache_key "$host$request_uri $cookie_user";
    proxy_temp_path  /var/cache/nginx/temp;
    proxy_ignore_headers Expires Cache-Control;
    proxy_cache_use_stale error timeout invalid_header http_502;
    proxy_cache_valid any 1d;

    # FastCGI cache
    fastcgi_cache_path /var/cache/nginx/micro levels=1:2 keys_zone=microcache:10m max_size=1024m inactive=30m;
    fastcgi_cache_key "$scheme$request_method$host$request_uri";
    fastcgi_cache_methods GET HEAD;
    fastcgi_cache_use_stale updating error timeout invalid_header http_500 http_503;
    fastcgi_ignore_headers Cache-Control Expires Set-Cookie;
    add_header X-FastCGI-Cache $upstream_cache_status;

    # Cache bypass
    map $http_cookie $no_cache {
        default 0;
        ~SESS 1;
        ~wordpress_logged_in 1;
        ~*remote_control.php 1;
    }

    # File cache (static assets)
    open_file_cache          max=10000 inactive=30s;
    open_file_cache_valid    60s;
    open_file_cache_min_uses 2;
    open_file_cache_errors   off;

    # Wildcard include
    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/conf.d/domains/*.conf;
}

 

Link to comment
Share on other sites

Do you have any panel or installed a Graphical server management ?

Did you run "top" and check what process is creating the load? Did you check the SSD health ?

I have a client with about 75K users daily  running on a centos 8 machine with 125K videos , script and database same server.

Xeon E3-1245v2 (4c/8th) - 32GB DDR3 1333 MHz - SoftRaid 2x480GB SSD

 

1533629459_Screenshot2021-06-23at06_43_17.thumb.jpg.f242e59519d5b2d62004f71683536634.jpg

 

Link to comment
Share on other sites

23 hours ago, xvids said:

I have Ubuntu 20.04:

I edited some Nginx and Apache config. Now L.A 5-6 because All time background task in process,

But now Database on MariaDB is working Slow, Please suggest me some info for DB, I have seen many answer on Stackweb that convert MyIsam to Innodb,

Intel Xeon-D 2141I - 8c/ 16t - 2.2GHz/ 3GHz

Memory : 64GB

Ubuntu: 20.04

[client]
port=3306
socket=/var/run/mysqld/mysqld.sock

[mysqld_safe]
socket=/var/run/mysqld/mysqld.sock
open_files_limit=4294967295

[mysqld]
user=mysql
pid-file=/var/run/mysqld/mysqld.pid
socket=/var/run/mysqld/mysqld.sock
port=3306
basedir=/usr
datadir=/var/lib/mysql
tmpdir=/tmp
lc-messages-dir=/usr/share/mysql
log_error=/var/log/mysql/error.log

symbolic-links=0
local-infile=0

skip-name-resolve
skip-external-locking
key_buffer_size = 1024M
max_allowed_packet = 1G
thread_stack = 384K
table_open_cache = 512
sort_buffer_size = 11M
read_buffer_size = 21M
read_rnd_buffer_size = 24M
myisam_sort_buffer_size = 512M
myisam_block_size = 5120
myisam_data_pointer_size = 7

key_cache_block_size = 8192
key_cache_division_limit = 60
key_cache_file_hash_size = 8192
key_cache_segments = 0

thread_cache_size = 256
query_cache_size= 300M
query_cache_type = 1
query_cache_limit = 256K
query_cache_min_res_unit = 2k

tmp_table_size= 256M
max_heap_table_size= 256M

#innodb_use_native_aio = 0
innodb_file_per_table

max_connections=2000
max_user_connections=500
wait_timeout=10
interactive_timeout=50
long_query_time=5

!includedir /etc/mysql/conf.d/
!includedir /etc/mysql/mariadb.conf.d/

Apache config:

 

# Global configuration
PidFile ${APACHE_PID_FILE}
Timeout 40
KeepAlive On
MaxKeepAliveRequests 3000
KeepAliveTimeout 5

<IfModule mpm_prefork_module>
    ServerLimit         256
    StartServers        10
    MinSpareServers     75
    MaxSpareServers     256
    MaxRequestWorkers   4500
    MaxConnectionsPerChild 10000
</IfModule>

<IfModule mpm_worker_module>
    ServerLimit            256
    StartServers           10
    MinSpareThreads        75
    MaxSpareThreads        256
    ThreadLimit            64
    ThreadsPerChild        32
    MaxRequestWorkers      4500
    MaxConnectionsPerChild 10000
</IfModule>

<IfModule mpm_event_module>
    StartServers        10
    MinSpareThreads     75
    MaxSpareThreads     256
    ThreadLimit         64
    ThreadsPerChild     32
    MaxRequestWorkers   4500
    MaxConnectionsPerChild 10000
</IfModule>

 

 

Nginx config:

 

 

# Server globals
user                    www-data;
worker_processes        24;
worker_rlimit_nofile    65535;
error_log               /var/log/nginx/error.log;
pid                     /var/run/nginx.pid;

include /etc/nginx/modules-enabled/*.conf;

# Worker config
events {
        worker_connections  4096;
        use                 epoll;
        multi_accept        off;
        accept_mutex        off;
}

http {
    # Main settings
    sendfile                        on;
        aio                             threads;
        directio                        8m;
    tcp_nopush                      on;
    tcp_nodelay                     on;
    client_header_timeout           180s;
    client_body_timeout             180s;
    client_header_buffer_size       2k;
    client_body_buffer_size         256k;
    client_max_body_size            256m;
    large_client_header_buffers     4 8k;
    send_timeout                    60s;
    keepalive_timeout               60s;
    keepalive_requests              20000;
    reset_timedout_connection       on;
    server_tokens                   off;
    server_name_in_redirect         off;
    server_names_hash_max_size      512;
    server_names_hash_bucket_size   512;
    charset                         utf-8;

    # FastCGI settings
    fastcgi_buffers                 4 256k;
    fastcgi_buffer_size             256k;
    fastcgi_busy_buffers_size       256k;
    fastcgi_temp_file_write_size    256k;
    fastcgi_connect_timeout         30s;
    fastcgi_read_timeout            300s;
    fastcgi_send_timeout            180s;
    fastcgi_cache_lock              on;
    fastcgi_cache_lock_timeout      5s;
    fastcgi_cache_background_update on;
    fastcgi_cache_revalidate        on;

    # Proxy settings
    proxy_redirect                  off;
    proxy_set_header                Host $host;
    proxy_set_header                X-Real-IP $remote_addr;
    proxy_set_header                X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_http_version              1.1;
    proxy_pass_header               Set-Cookie;
    proxy_buffers                   32 4k;
    proxy_connect_timeout           30s;
    proxy_read_timeout              300s;
    proxy_send_timeout              180s;

    # Log format
    log_format  main    '$remote_addr - $remote_user [$time_local] $request '
                        '"$status" $body_bytes_sent "$http_referer" '
                        '"$http_user_agent" "$http_x_forwarded_for"';
    log_format  bytes   '$body_bytes_sent';
    log_not_found off;
    access_log off;

    # Mime settings
    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    # Compression
    gzip                on;
    gzip_static         on;
    gzip_vary           on;
    gzip_comp_level     6;
    gzip_min_length     1024;
    gzip_buffers        16 8k;
    gzip_http_version   1.1;
    gzip_types          text/plain text/css text/javascript text/js text/xml application/json application/javascript application/x-javascript application/xml application/xml+rss applicatio>
    gzip_proxied        any;
    gzip_disable        "MSIE [1-6]\.";

    # Cloudflare https://www.cloudflare.com/ips
    set_real_ip_from 103.21.244.0/22;
    # SSL PCI compliance
    ssl_session_cache   shared:SSL:20m;
    ssl_session_timeout 60m;
    ssl_buffer_size     1400;
    ssl_protocols       TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers on;
    ssl_ciphers         
    ssl_dhparam         /etc/ssl/dhparam.pem;
    ssl_ecdh_curve      secp384r1;
    ssl_session_tickets off;
    resolver 213.186.33.99  valid=300s ipv6=off;
    resolver_timeout    5s;

    # Error pages
    error_page 403 /error/404.html;
    error_page 404 /error/404.html;
    error_page 410 /error/410.html;
    error_page 500 501 502 503 504 505 /error/50x.html;

    # Proxy cache
    proxy_cache_path /var/cache/nginx levels=2 keys_zone=cache:10m inactive=60m max_size=1024m;
    proxy_cache_key "$host$request_uri $cookie_user";
    proxy_temp_path  /var/cache/nginx/temp;
    proxy_ignore_headers Expires Cache-Control;
    proxy_cache_use_stale error timeout invalid_header http_502;
    proxy_cache_valid any 1d;

    # FastCGI cache
    fastcgi_cache_path /var/cache/nginx/micro levels=1:2 keys_zone=microcache:10m max_size=1024m inactive=30m;
    fastcgi_cache_key "$scheme$request_method$host$request_uri";
    fastcgi_cache_methods GET HEAD;
    fastcgi_cache_use_stale updating error timeout invalid_header http_500 http_503;
    fastcgi_ignore_headers Cache-Control Expires Set-Cookie;
    add_header X-FastCGI-Cache $upstream_cache_status;

    # Cache bypass
    map $http_cookie $no_cache {
        default 0;
        ~SESS 1;
        ~wordpress_logged_in 1;
        ~*remote_control.php 1;
    }

    # File cache (static assets)
    open_file_cache          max=10000 inactive=30s;
    open_file_cache_valid    60s;
    open_file_cache_min_uses 2;
    open_file_cache_errors   off;

    # Wildcard include
    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/conf.d/domains/*.conf;
}

 

I think you overconfig sql...Sent pm 

Link to comment
Share on other sites

  • 2 weeks later...
On 6/23/2021 at 8:51 AM, phseven said:

Do you have any panel or installed a Graphical server management ?

Yes I have Installed HestiaCP with Nginx Apache, but not PHP-FPM, because If enable PHP-FPM then some errors with .htaccess, I don't remmeber that,

maybe the issue is "Folder allows PHP execution"

 

On 6/23/2021 at 8:51 AM, phseven said:

Did you run "top" and check what process is creating the load?

Yes by run TOP Its displaying, Nginx, youtube-dl, Apache,

 

On 6/23/2021 at 8:51 AM, phseven said:

Did you check the SSD health ?

Yes looks normal,

Link to comment
Share on other sites

On 7/2/2021 at 5:44 PM, xvids said:

Yes I have Installed HestiaCP with Nginx Apache, but not PHP-FPM, because If enable PHP-FPM then some errors with .htaccess, I don't remmeber that,

maybe the issue is "Folder allows PHP execution"

 

Yes by run TOP Its displaying, Nginx, youtube-dl, Apache,

 

Yes looks normal,

 

Hey,

 

what mpm php are you using? prefork? event? worker? 

 

Link to comment
Share on other sites

On the server where you have the database the top process should be mysql followed by nginx or apache.

Server running script should be nginx or apache (httpd) whatever you have installed.

I use the tool gotop for monitoring servers .

My wild guess is that you have a clog somewhere in the I/O process that spikes the server load.

 

 

 

Link to comment
Share on other sites

On 7/4/2021 at 11:07 PM, elpadre said:

what mpm php are you using? prefork? event? worker? 

prefork PHP7.4

 

On 7/5/2021 at 10:30 AM, phseven said:

On the server where you have the database the top process should be mysql followed by nginx or apache.

https://prnt.sc/194h874

 

On 7/5/2021 at 10:30 AM, phseven said:

Server running script should be nginx or apache (httpd) whatever you have installed.

Both Installed Nginx & Apache.

On 7/5/2021 at 10:30 AM, phseven said:

My wild guess is that you have a clog somewhere in the I/O process that spikes the server load.

Yes This maybe, in RAID, I think raid not configure well.

Link to comment
Share on other sites

On 6/21/2021 at 9:48 AM, Tech Support said:

No, this is not officially supported.

Hello TechSupport, just to clarify the load balancing question: is Load Balancing forbidden/restricted by the KVS app or license? Or are we free to setup multiple app servers serving the same domain and try to implement it on our own?

Link to comment
Share on other sites

1 hour ago, Lupu said:

Hello TechSupport, just to clarify the load balancing question: is Load Balancing forbidden/restricted by the KVS app or license? Or are we free to setup multiple app servers serving the same domain and try to implement it on our own?

Hello, yes, setting up multiple KVS installations for the same domain is not allowed by KVS license terms. And thus we will not be to provide any support for such configurations.

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...