Z-Push Java Shim - Administrator Installation Guide

This guide provides system administrators with complete instructions for installing, configuring, and maintaining the Z-Push Java Shim that dramatically improves ActiveSync performance and fixes critical bugs.

Note

I have been testing this shim since August 2025 on my iPhone and still use it to this day, and throughout that time it has performed as well as—if not faster than—the Zimbra Network Edition implementation which is also on my iPhone. That still surprises me, but the results have been consistent.

This project originally existed because Z-Push and the FOSS 2FA implementation had an authentication issue related to application passwords, which prevented ActiveSync from working correctly. That issue has since been resolved, and Z-Push now works properly with FOSS 2FA. As a result, this shim is no longer necessary, and I am discontinuing active development.

One pleasant discovery from this project was how fast Z-Push performs on our servers. If you're evaluating ActiveSync support, Z-Push is still worth considering. The included nginx Docker container can run stock Z-Push with a small change to the environment file, allowing you to use it directly without the shim.

Safe Deploy/Undeploy (At a Glance)

• Single artifact: installs exactly one file — /opt/zimbra/lib/ext/zpush-shim/zpush-shim.jar.
• Isolated: runs as a Zimbra extension (no core JARs replaced; no schema/config changes).
• Clean enable/disable:
  • Deploy: ./deploy-shim.sh --deploy then su - zimbra -c 'zmmailboxdctl restart'
  • Undeploy: ./deploy-shim.sh --undeploy then restart (moves to /opt/zimbra/lib/ext-disabled/...)
• Zero‑risk preview: ./deploy-shim.sh --plan (shows exactly what --deploy would do)
• Quick status: ./deploy-shim.sh --status (paths, endpoint ping, recent log lines)
• Verify endpoint: ./deploy-shim.sh --verify or make verify-ping

Table of Contents

1. Overview
2. Prerequisites
3. Quick Installation
4. Manual Installation
5. Configuration
6. Verification
7. Performance Benefits
8. Troubleshooting
9. Maintenance
10. Uninstallation

Overview

One-file install, instant rollback, zero surprises:

• One JAR: /opt/zimbra/lib/ext/zpush-shim/zpush-shim.jar (no core JARs touched)
• Instant rollback: ./deploy-shim.sh --undeploy + restart
• Loopback-only IMAP for app passwords; external IMAP can remain blocked

What This Solves

The Z-Push Java Shim addresses critical performance and reliability issues in Z-Push Release 74:

Critical Bug Fixed:

• Folder names with spaces: Z-Push Release 74 REST API returns 404 errors for folders like "0-Zimbra Security" or "My Projects"
• URL encoding issues: Complex folder names cause sync failures

Performance Improvements:

• 20-60x faster folder hierarchy loading (15,000 folders: 15 seconds → 200ms)
• 10x less memory usage during sync operations
• Instant folder operations even with complex folder structures

Enhanced Features:

• Proper 2FA support: Application passwords work correctly with EAS authentication context
• Automatic fallback: If shim fails, seamlessly falls back to original SOAP method
• Production reliability: Comprehensive error handling and logging

How It Works

BEFORE (SOAP):
Mobile Device → Z-Push → SOAP/XML → HTTP → Zimbra SOAP Handler → Internal APIs
                          ↑ Slow, buggy with spaces, high memory usage

AFTER (Java Shim):
Mobile Device → Z-Push → JSON → HTTP → Java Shim → Zimbra Internal APIs (direct)
                          ↑ Fast, reliable, low memory usage

The Java Shim runs inside Zimbra's existing Jetty servlet container and provides a lightweight HTTP/JSON interface to Zimbra's internal APIs, bypassing the SOAP overhead entirely.

Prerequisites

System Requirements

• Zimbra Collaboration Suite: OSE or Network Edition
• Z-Push 2.7+: Automatically installed via Docker container
• Java 11+: Usually included with Zimbra
• Apache Ant: For building (Zimbra's official build system)
• Docker and Docker Compose: For Z-Push container deployment

Supported Configurations

• Zimbra OSE: 10.0+ (fully supported - tested with 10.0 and 10.1)
• Zimbra Network Edition: 10.0+ (compatible - already has ActiveSync)
• Z-Push: 2.7.0+ (tested with Release 74)
• Operating Systems: Zimbra-supported Linux distributions (CentOS, RHEL, Ubuntu LTS)

Network Requirements

• Internal communication: Shim communicates with Zimbra on localhost:8080
• No external ports: Uses existing Zimbra infrastructure
• SSL: Inherits Zimbra's SSL configuration

Quick Installation

Option 1: Automated Installation (Recommended)

# 1. Navigate to the shim directory
cd /path/to/zpushshim

# 2. Run complete installation
sudo ./deploy-shim.sh --all

# 3. Configure Z-Push
sudo nano /path/to/z-push/config.php
# Add: define('ZIMBRA_USE_JAVA_SHIM', true);

# 4. Test
curl -X POST -d "action=ping" http://localhost:8080/service/extension/zpush-shim

Option 2: Using Pre-built JAR (manual extension deploy)

# 1. Copy JAR to Zimbra extension directory
sudo mkdir -p /opt/zimbra/lib/ext/zpush-shim
sudo cp dist/zpush-shim.jar /opt/zimbra/lib/ext/zpush-shim/zpush-shim.jar
sudo chown root:root /opt/zimbra/lib/ext/zpush-shim/zpush-shim.jar
sudo chmod 444 /opt/zimbra/lib/ext/zpush-shim/zpush-shim.jar

# 2. Restart mailboxd to load the extension
su - zimbra -c "zmmailboxdctl restart"

# 3. Configure and test as above

Manual Installation

Step 1: Build the Java Shim (if needed)

# Install build tools (if not present)
# CentOS/RHEL:
sudo yum install -y java-11-openjdk-devel ant

# Ubuntu/Debian:
sudo apt-get install -y openjdk-11-jdk ant

# Build the shim
cd /path/to/zpushshim
ant clean jar

# Verify build
ls -la dist/zpush-shim.jar

Step 2: Deploy to Zimbra

# Find Zimbra installation
ZIMBRA_HOME=$(su - zimbra -c 'echo $ZIMBRA_HOME')
echo "Zimbra home: $ZIMBRA_HOME"

# Deploy as an extension (recommended)
sudo ./deploy-shim.sh --deploy

# Verify deployment
ls -la /opt/zimbra/lib/ext/zpush-shim/zpush-shim.jar

Step 3: Restart Zimbra Services

# Stop Zimbra
su - zimbra -c "zmcontrol stop"

# Verify all services are stopped
su - zimbra -c "zmcontrol status"

# Start Zimbra
su - zimbra -c "zmcontrol start"

# Verify all services are running
su - zimbra -c "zmcontrol status"

Step 4: Configure Z-Push

# Locate Z-Push configuration
Z_PUSH_CONFIG="/usr/share/z-push/src/config.php"  # Adjust path as needed

# Backup original configuration
sudo cp $Z_PUSH_CONFIG $Z_PUSH_CONFIG.backup.$(date +%Y%m%d)

# Add shim configuration
sudo tee -a $Z_PUSH_CONFIG << EOF

# Z-Push Java Shim Configuration
define('ZIMBRA_USE_JAVA_SHIM', true);
define('ZIMBRA_SHIM_URL', 'http://localhost:8080/service/extension/zpush-shim');
define('ZIMBRA_SHIM_TIMEOUT', 30);
EOF

Configuration

Basic Configuration

Essential Settings in Z-Push config.php:

# Enable Java Shim
define('ZIMBRA_USE_JAVA_SHIM', true);

# Shim URL (default is usually correct)
define('ZIMBRA_SHIM_URL', 'http://localhost:8080/service/extension/zpush-shim');

# Timeout for shim calls (seconds)
define('ZIMBRA_SHIM_TIMEOUT', 30);

Advanced Configuration

Performance Tuning:

# Keep REST API enabled (shim bypasses its bugs)
define('ZIMBRA_GETMESSAGELIST_USE_REST_API', true);

# Enable virtual folders for better multi-folder support
define('ZIMBRA_VIRTUAL_CONTACTS', true);
define('ZIMBRA_VIRTUAL_APPOINTMENTS', true);
define('ZIMBRA_VIRTUAL_TASKS', true);
define('ZIMBRA_VIRTUAL_NOTES', true);

# Optimize sync window
define('ZIMBRA_SYNC_WINDOW_DAYS', 90);

Debugging Configuration:

# Enable debug logging to see shim activity
define('LOGLEVEL', LOGLEVEL_DEBUG);
define('LOGFILE', '/var/log/z-push/z-push.log');

Zimbra-Specific Configuration

For deployments with many folders:

# Increase Zimbra JVM heap if needed
su - zimbra -c "zmlocalconfig -e zimbra_server_jvm_heap_size=2048m"
su - zimbra -c "zmcontrol restart"

For high-availability setups:

# If Z-Push runs on different server than Zimbra
define('ZIMBRA_SHIM_URL', 'http://zimbra-server.internal:8080/service/extension/zpush-shim');

Verification

Step 1: Verify Shim Deployment

# Check JAR file is present
ls -la /opt/zimbra/lib/ext/zpush-shim/zpush-shim.jar

# Check file permissions
# Should be: zimbra:zimbra with 644 permissions

Step 2: Test Shim Endpoint

# Test ping endpoint
curl -X POST -d "action=ping" http://localhost:8080/service/extension/zpush-shim

# Expected response:
# {"status":"ok","version":"1.0.0","timestamp":1234567890}

# Test with verbose output
curl -v -X POST -d "action=ping" http://localhost:8080/service/extension/zpush-shim

Step 3: Verify Z-Push Integration

# Check Z-Push logs for shim initialization
tail -f /var/log/z-push/z-push.log | grep -i shim

# Look for messages like:
# [DEBUG] Zimbra->_initializeShim(): Java Shim enabled at http://localhost:8080/service/extension/zpush-shim

Step 4: Test with Mobile Device

1. Configure ActiveSync Account:

   • Server: your-zimbra-server.com/z-push
   • Username: Zimbra username
   • Password: Zimbra password (or app password if 2FA enabled)

2. Verify Sync:

   • Email synchronization should be noticeably faster
   • Folders with spaces should sync without errors
   • Check device shows all folders correctly

3. Monitor Performance:

   # Watch Z-Push logs during sync
   tail -f /var/log/z-push/z-push.log | grep -E "shim|Shim"

Step 5: Performance Verification

Test folder hierarchy loading:

# Time a folder sync operation (monitor logs)
# Before: 5-15 seconds for large hierarchies
# After: 50-200ms with shim

# Check memory usage during sync
ps aux | grep -E "httpd|apache|nginx" | head -5

Performance Benefits

Measured Improvements

Operation	Before (SOAP)	After (Shim)	Improvement
15,000 folders load	5-15 seconds	50-200ms	30-300x
1,000 messages list	2-10 seconds	50-200ms	10-200x
Folder with spaces	❌ 404 errors	✅ Works	Bug fixed
Memory per sync	200-500MB	20-50MB	10x less
Authentication w/2FA	❌ Often fails	✅ Reliable	Bug fixed

Real-World Impact

Small Organizations (< 1,000 folders):

• Initial sync: 30 seconds → 3 seconds
• Folder operations: Near-instant
• Fewer timeout errors

Large Organizations (> 10,000 folders):

• Initial sync: 10+ minutes → 30 seconds
• Dramatic reduction in server load
• Supports more concurrent users

Users with Complex Folder Names:

• Folders like "My Documents 2024" now work perfectly
• No more mysterious sync failures
• Reliable multi-folder sync

Troubleshooting

Common Issues

Authentication to Shim

Symptoms:

• 401 from shim on authenticate while SOAP works
• POSTs with cookies fail inconsistently

Solutions:

• When using cookies, also send CSRF if present:
  • The harness auto-adds X-Zimbra-Csrf-Token if SOAP returns it.
• Alternatively, authenticate using the header token:
  • Send X-Zimbra-Auth-Token: <ZM_AUTH_TOKEN> to the shim with action=authenticate.
• Helper scripts during manual testing:
  • optionA.sh: performs SOAP login and authenticates using the header token in one shot.
  • j1.sh: cookie-based authenticate (uses c.txt and optional CSRF from soap.json).
  • j2.sh: username/password authenticate (URL-encoded; prompts if password missing).

1. Shim Not Responding

Symptoms:

• Z-Push logs show "Shim call failed, falling back to SOAP"
• Curl test to shim endpoint fails

Solutions:

# Check Zimbra services
su - zimbra -c "zmcontrol status"

# Check if JAR is properly deployed
ls -la /opt/zimbra/lib/ext/zpush-shim/zpush-shim.jar

# Check Zimbra logs for startup errors
tail -f /opt/zimbra/log/mailbox.log | grep -i "zpush\|shim\|error"

# Restart Zimbra if needed
su - zimbra -c "zmcontrol restart"

2. Authentication Failures

Symptoms:

• Mobile devices can't sync
• Shim returns authentication errors

Solutions:

# Test Zimbra authentication directly
curl -k -d "loginOp=login&username=testuser&password=testpass" \
     https://your-zimbra-server.com/service/preauth

# For 2FA users, ensure app passwords are used
# Check Z-Push logs for auth details
grep -i "auth" /var/log/z-push/z-push.log | tail -10

3. Folders Still Not Syncing

Symptoms:

• Some folders with spaces still don't sync
• Shim appears to be working

Solutions:

# Verify shim is actually being used
grep "Java Shim" /var/log/z-push/z-push.log

# Test specific folder operations
curl -X POST -d "action=getfolders&username=test&password=test" \
     http://localhost:8080/service/extension/zpush-shim

# Check folder names for special characters
# Some characters may need additional handling

4. Performance Not Improved

Symptoms:

• Sync times haven't improved significantly
• High memory usage continues

Solutions:

# Verify shim is being used (not falling back to SOAP)
grep -c "falling back to SOAP" /var/log/z-push/z-push.log

# If count > 0, shim calls are failing
# Check shim configuration and connectivity

# Monitor actual shim usage
tail -f /var/log/z-push/z-push.log | grep -E "Successful shim call|Shim call failed"

Debug Mode

Enable comprehensive debugging:

# In Z-Push config.php
define('LOGLEVEL', LOGLEVEL_DEBUG);
define('LOGFILE', '/var/log/z-push/debug.log');

# Create debug log directory
sudo mkdir -p /var/log/z-push
sudo chown www-data:www-data /var/log/z-push

Monitor debug output:

# Watch all Z-Push activity
tail -f /var/log/z-push/debug.log

# Filter for shim-related activity
tail -f /var/log/z-push/debug.log | grep -i shim

# Monitor Zimbra side
tail -f /opt/zimbra/log/mailbox.log | grep -i zpush

Network Troubleshooting

Test network connectivity:

# Test from Z-Push server to Zimbra (if separate)
telnet zimbra-server 8080

# Test HTTP connectivity
curl -I http://localhost:8080/service/

# Check firewall rules
iptables -L | grep 8080

Maintenance

Regular Tasks

Daily Monitoring

# Check for errors in logs
grep -i error /var/log/z-push/z-push.log | tail -10

# Monitor performance
grep "took.*ms" /var/log/z-push/z-push.log | tail -5

# Check Zimbra service health
su - zimbra -c "zmcontrol status"

Weekly Tasks

# Rotate Z-Push logs
logrotate -f /etc/logrotate.d/z-push

# Check disk space
df -h /opt/zimbra /var/log

# Review shim usage statistics
grep -c "Successful shim call" /var/log/z-push/z-push.log
grep -c "falling back to SOAP" /var/log/z-push/z-push.log

Monthly Tasks

# Update Java shim if new version available
./deploy-shim.sh --build --deploy --restart

# Review performance metrics
# Compare sync times before/after shim deployment

# Check for Zimbra updates that might affect shim
su - zimbra -c "zmcontrol -v"

Backup and Recovery

Backup Configuration:

# Backup Z-Push configuration
cp /usr/share/z-push/src/config.php /backup/z-push-config.$(date +%Y%m%d).php

# Backup shim JAR
cp /opt/zimbra/lib/ext/zpush-shim/zpush-shim.jar \
   /backup/zpush-shim-$(date +%Y%m%d).jar

Recovery Procedure:

# Restore from backup
cp /backup/zpush-shim-YYYYMMDD.jar /opt/zimbra/lib/ext/zpush-shim/zpush-shim.jar
cp /backup/z-push-config-YYYYMMDD.php /usr/share/z-push/src/config.php

# Restart services
su - zimbra -c "zmcontrol restart"

Updates and Upgrades

Updating the Java Shim:

# 1. Stop Z-Push (optional - graceful update)
# 2. Build new version
mvn clean package

# 3. Deploy
./deploy-shim.sh --deploy --restart

# 4. Verify
curl -X POST -d "action=ping" http://localhost:8080/service/extension/zpush-shim

Updating Zimbra:

# Before Zimbra update:
# 1. Backup shim JAR
cp /opt/zimbra/lib/ext/zpush-shim/zpush-shim.jar /backup/

# After Zimbra update:
# 1. Redeploy shim JAR
cp /backup/zpush-shim-1.0.0.jar /opt/zimbra/lib/ext/zpush-shim/zpush-shim.jar
chown root:root /opt/zimbra/lib/ext/zpush-shim/zpush-shim.jar
chmod 444 /opt/zimbra/lib/ext/zpush-shim/zpush-shim.jar

# 2. Restart and verify
su - zimbra -c "zmcontrol restart"

Uninstallation

Complete Removal

Step 1: Disable Shim in Z-Push

# Edit Z-Push config
sudo nano /usr/share/z-push/src/config.php

# Comment out or change:
# define('ZIMBRA_USE_JAVA_SHIM', false);

Step 2: Remove JAR File

# If deployed as an extension (recommended):
sudo ./deploy-shim.sh --undeploy
su - zimbra -c 'zmmailboxdctl restart'

# If you manually copied the JAR elsewhere, remove it accordingly and restart mailboxd.

Note: Do not leave backup directories under /opt/zimbra/lib/ext. Zimbra will attempt to load all subdirectories there. Our --undeploy moves the extension to /opt/zimbra/lib/ext-disabled/ to avoid accidental loading.

Step 3: Verify Fallback

# Check Z-Push logs for SOAP usage
tail -f /var/log/z-push/z-push.log | grep -i soap

# Test mobile device sync
# Should continue working via SOAP

Partial Disable (Keep JAR, Disable Feature)

# Just disable in configuration
sudo sed -i 's/ZIMBRA_USE_JAVA_SHIM.*true/ZIMBRA_USE_JAVA_SHIM", false/' \
    /usr/share/z-push/src/config.php

# No restart needed - change takes effect immediately

Summary

The Z-Push Java Shim provides dramatic performance improvements and critical bug fixes while maintaining full compatibility with existing Z-Push installations. The automatic fallback mechanism ensures reliable operation even if the shim encounters issues.

Key Benefits:

• ✅ 20-60x performance improvement
• ✅ Fixes folder space bug (Z-Push Release 74 issue)
• ✅ Proper 2FA/app password support
• ✅ Automatic SOAP fallback for reliability
• ✅ No mobile device changes needed
• ✅ Works with existing Z-Push installations

Administrative Overhead:

• Minimal: Uses existing Zimbra infrastructure
• Reliable: Comprehensive error handling and logging
• Maintainable: Standard Java deployment model

Permissions

• Extension directory ownership: /opt/zimbra/lib/ext is root:root (755). Deployed JARs are typically root:root with mode 444 (read-only).
• Deploy/Undeploy: Run as root (writes under /opt/zimbra/lib/ext).
• Restart services: Run as zimbra (e.g., su - zimbra -c "zmmailboxdctl restart").
• Our deploy script sets:
  • Directory: root:root 755
  • JAR: root:root 444
  • Then prompts to restart mailboxd as the zimbra user.

For most installations, the shim provides immediate and significant improvements with minimal risk and maintenance overhead.

License

This project is licensed under the MIT License.

/*
 * Copyright (c) 2025 Z-Push Zimbra Shim contributors
 * Licensed under the MIT License. See LICENSE file for details.
 */

See the LICENSE file at the repository root for the full text.