email-sync-oauth2/README.md

# Email Sync OAuth2

Dockerized multi-account IMAP synchronization tool with **OAuth2 support** for Office 365, Hotmail, Outlook.com, and Gmail.

## Problem This Solves

Microsoft retired **Basic Authentication** for Exchange Online (IMAP/SMTP), and Google is pushing XOAUTH2 as the only authentication method. This breaks traditional IMAP sync tools that rely on username/password.

**Mailcow's built-in sync job** and many other tools don't support OAuth2, making it impossible to migrate emails from Hotmail/Outlook.com/Gmail into self-hosted mail servers.

This project solves that problem by combining:
- **[imapsync](https://imapsync.lamiral.info/)** — battle-tested IMAP migration tool
- **imapsync's OAuth2 helper** — token authorization and refresh for Microsoft and Google accounts
- **Docker** — portable, reproducible, easy to deploy

## Features

| Feature | Status |
|---------|--------|
| Office 365 / Hotmail / Outlook.com sync | ✅ OAuth2 |
| Gmail / Google Workspace sync | ✅ OAuth2 |
| Any IMAP server (password auth) | ✅ |
| Multiple accounts in one container | ✅ Unlimited |
| Incremental sync (only new emails) | ✅ |
| Folder structure preservation | ✅ |
| Scheduled sync via interval | ✅ |
| Healthcheck endpoint | ✅ |
| One-shot sync mode | ✅ |
| Dry run mode | ✅ |
| Delete propagation (`--delete2`) | ✅ |
| Custom imapsync flags per account | ✅ |
| Persistent token storage | ✅ |
| Automatic token refresh | ✅ |
| SMTP sync notifications | ✅ Optional |
| Interval or cron scheduling | ✅ |

## Authentication Requirements

| Provider | Auth Method | Token Lifetime | Auto-Refresh |
|----------|-------------|----------------|--------------|
| **Hotmail / Outlook.com / Personal Microsoft** | OAuth2 | 90 days (hard limit) | ✅ until day 90 |
| **Office 365 / Microsoft 365 (Work/School)** | OAuth2 | 90 days (configurable by admin) | ✅ indefinitely |
| **Gmail / Google Workspace** | OAuth2 | 7 days (testing) / 90 days (verified) | ✅ |
| **Other IMAP** | Password | N/A | N/A |

> ⚠️ **Personal Microsoft accounts** (hotmail.com, outlook.com, live.com) have a hard 90-day refresh token limit. You must reauthorize every ~80 days. Work/school accounts can have indefinite tokens if configured by Azure AD admin.

---

## Quick Start

### 1. Clone and configure

```bash
git clone https://github.com/yourusername/email-sync-oauth2.git
cd email-sync-oauth2
cp .env.example .env
nano .env   # edit your settings
```

### 2. First-time OAuth authorization (requires browser)

Run the container interactively for each account:

```bash
# For Office 365 / Hotmail (Account 1)
docker compose run --rm imapsync authorize 1
# → Paste URL in browser, log in to Microsoft, click "Allow"
# → Token saved to ./tokens/user1_hotmail.token

# For Account 2
docker compose run --rm imapsync authorize 2
# → Same process for this account
```

### 3. Start syncing

```bash
docker compose up -d
```

### 4. Monitor

```bash
docker logs -f email-sync-oauth2
tail -f logs/account1.log
```

---

## Configuration

All configuration is done through environment variables. Edit `.env` or pass directly to `docker-compose.yml`.

### Global Variables

| Variable | Default | Description |
|----------|---------|-------------|
| `NUM_ACCOUNTS` | `1` | Number of accounts to sync |
| `SYNC_INTERVAL` | `21600` (6h) | Seconds between sync cycles |
| `SYNC_CRON` | empty | Optional five-field cron expression; overrides `SYNC_INTERVAL` |
| `ONCE` | `false` | Run once and exit (no daemon) |
| `TOKEN_DIR` | `/tokens` | Directory inside container for token files |
| `LOG_DIR` | `/var/log/imapsync` | Directory inside container for log files |

### Per-Account Variables (ACCOUNT_1_, ACCOUNT_2_, ...)

| Variable | Required | Description |
|----------|----------|-------------|
| `ACCOUNT_N_SOURCE_HOST` | ✅ | Source IMAP server (e.g., `outlook.office365.com`) |
| `ACCOUNT_N_SOURCE_USER` | ✅ | Source email address |
| `ACCOUNT_N_SOURCE_AUTH` | ✅ | `oauth2` or `password` |
| `ACCOUNT_N_SOURCE_PROVIDER` | If OAuth2 | `office365`, `hotmail`, `outlook`, `gmail`, `google` |
| `ACCOUNT_N_SOURCE_TOKEN` | If OAuth2 | Path to token file (e.g., `/tokens/account1.token`) |
| `ACCOUNT_N_SOURCE_PASS` | If password | Source password |
| `ACCOUNT_N_DEST_HOST` | ✅ | Destination IMAP server (e.g., `mail.hoelee.com`) |
| `ACCOUNT_N_DEST_USER` | ✅ | Destination email/username |
| `ACCOUNT_N_DEST_PASS` | ✅ | Destination password |
| `ACCOUNT_N_DELETE` | No | `true` = delete emails on destination not on source |
| `ACCOUNT_N_DRY_RUN` | No | `true` = log only, don't actually sync |
| `ACCOUNT_N_EXTRA_OPTS` | No | Additional imapsync flags (optional) |

### Password authentication is still supported

OAuth2 is required only for providers that require it. Any normal IMAP source
can still use a password by setting `ACCOUNT_N_SOURCE_AUTH=password` and
`ACCOUNT_N_SOURCE_PASS`. The destination currently uses password
authentication for every provider.

```env
ACCOUNT_3_SOURCE_HOST=imap.example.com
ACCOUNT_3_SOURCE_USER=[email protected]
ACCOUNT_3_SOURCE_AUTH=password
ACCOUNT_3_SOURCE_PASS=source_password

ACCOUNT_3_DEST_HOST=mail.example.net
ACCOUNT_3_DEST_USER=[email protected]
ACCOUNT_3_DEST_PASS=destination_password
```

### Scheduling

By default, a batch runs immediately when the container starts and then every
six hours (`SYNC_INTERVAL=21600`). Set the interval in seconds to use a
different fixed frequency:

```env
SYNC_INTERVAL=3600
SYNC_CRON=
```

For calendar-based scheduling, set `SYNC_CRON` to a standard five-field cron
expression. It overrides `SYNC_INTERVAL`.

```env
# Every six hours, on the hour
SYNC_CRON=0 */6 * * *

# Every day at 02:30
# SYNC_CRON=30 2 * * *
```

All `ACCOUNT_N_*` variables in `.env` are passed to the container. To add an
account, copy an account block, use the next number, and increase
`NUM_ACCOUNTS`—there is no two-account Compose limit.

### SMTP notifications

Notifications are off by default. Set `NOTIFY_MODE` to one of these values:

| Value | When an email is sent |
|---|---|
| `batch_success` | Once after every batch where every configured account succeeds |
| `account_success` | Once for each account that succeeds |
| `failure` | Once after a batch where one or more accounts fail |

Configure SMTP whenever notifications are enabled:

```env
NOTIFY_MODE=batch_success
SMTP_HOST=smtp.example.com
SMTP_PORT=587
SMTP_USER=[email protected]
SMTP_PASS=your_smtp_password
SMTP_FROM=[email protected]
SMTP_TO=[email protected],[email protected]
SMTP_TLS=starttls
```

`SMTP_TLS` accepts `starttls` (normally port 587), `tls` (implicit TLS,
normally port 465), or `none`. If a notification cannot be delivered, the
container writes the reason to `logs/notification.log`, logs the failure to the
container output, and becomes unhealthy until a notification sends
successfully.

---

## Complete Examples

### Example 1: Hotmail → Mailcow (OAuth2)

```env
NUM_ACCOUNTS=1
SYNC_INTERVAL=21600

ACCOUNT_1_SOURCE_HOST=outlook.office365.com
ACCOUNT_1_SOURCE_USER=[email protected]
ACCOUNT_1_SOURCE_AUTH=oauth2
ACCOUNT_1_SOURCE_PROVIDER=office365
ACCOUNT_1_SOURCE_TOKEN=/tokens/you_hotmail.token

ACCOUNT_1_DEST_HOST=mail.hoelee.com
ACCOUNT_1_DEST_USER=[email protected]
ACCOUNT_1_DEST_PASS=your_mailcow_password
```

### Example 2: Gmail → Mailcow (OAuth2)

```env
ACCOUNT_1_SOURCE_HOST=imap.gmail.com
ACCOUNT_1_SOURCE_USER=[email protected]
ACCOUNT_1_SOURCE_AUTH=oauth2
ACCOUNT_1_SOURCE_PROVIDER=gmail
ACCOUNT_1_SOURCE_TOKEN=/tokens/you_gmail.token
```

### Example 3: Multiple accounts

```env
NUM_ACCOUNTS=3

# Account 1: Hotmail -> Mailcow
ACCOUNT_1_SOURCE_HOST=outlook.office365.com
ACCOUNT_1_SOURCE_USER=[email protected]
ACCOUNT_1_SOURCE_AUTH=oauth2
ACCOUNT_1_SOURCE_PROVIDER=office365
ACCOUNT_1_SOURCE_TOKEN=/tokens/alice_hotmail.token
ACCOUNT_1_DEST_HOST=mail.hoelee.com
ACCOUNT_1_DEST_USER=[email protected]
ACCOUNT_1_DEST_PASS=password1

# Account 2: Gmail -> Mailcow
ACCOUNT_2_SOURCE_HOST=imap.gmail.com
ACCOUNT_2_SOURCE_USER=[email protected]
ACCOUNT_2_SOURCE_AUTH=oauth2
ACCOUNT_2_SOURCE_PROVIDER=gmail
ACCOUNT_2_SOURCE_TOKEN=/tokens/bob_gmail.token
ACCOUNT_2_DEST_HOST=mail.hoelee.com
ACCOUNT_2_DEST_USER=[email protected]
ACCOUNT_2_DEST_PASS=password2

# Account 3: Legacy IMAP -> Mailcow (password auth)
ACCOUNT_3_SOURCE_HOST=imap.oldprovider.com
ACCOUNT_3_SOURCE_USER=[email protected]
ACCOUNT_3_SOURCE_AUTH=password
ACCOUNT_3_SOURCE_PASS=oldpassword
ACCOUNT_3_DEST_HOST=mail.hoelee.com
ACCOUNT_3_DEST_USER=[email protected]
ACCOUNT_3_DEST_PASS=password3
```

### Example 4: One-shot sync (cron-friendly)

```bash
# .env
ONCE=true
NUM_ACCOUNTS=1

# Add to crontab:
# 0 */6 * * * cd /opt/email-sync-oauth2 && docker compose up --abort-on-container-exit
```

---

## Makefile Commands (Optional)

A `Makefile` is included for convenience:

```bash
# Build and start
make build
make up

# Check status and health
make status
make health

# Authorize an account (requires browser)
make authorize ACCOUNT=1

# Force sync now
make sync ACCOUNT=1

# Force a batch for every configured account
make sync-all

# Follow logs
make logs

# Enter container shell
make shell

# Run once and exit (one-shot mode)
make once

# Clean up
make clean
```

---

## Manual Operations

### Force sync a specific account immediately

```bash
docker compose exec imapsync sync.sh sync 1
```

### Reauthorize OAuth tokens

```bash
docker compose run --rm imapsync authorize 1
```

### Enter container shell

```bash
docker compose exec imapsync sync.sh shell
```

### Run imapsync manually with custom flags

```bash
docker compose exec imapsync sync.sh imapsync \
  --host1 outlook.office365.com --user1 [email protected] \
  --oauthaccesstoken1 /tokens/you_hotmail.token \
  --host2 mail.hoelee.com --user2 [email protected] \
  --password2 yourpassword \
  --ssl1 --ssl2 --dry
```

---

## Monitoring & Healthcheck

The container includes a Docker **healthcheck** based on a heartbeat mechanism:

- A background process writes a heartbeat file every **60 seconds**
- Docker checks if the heartbeat is less than **5 minutes old**
- If no heartbeat, the container is marked unhealthy and can be auto-restarted by orchestrators

It also becomes unhealthy if a configured SMTP notification fails to send. The
reason is logged in `logs/notification.log` and in the container output.

The heartbeat confirms that the sync daemon is running; it does not mean every
mailbox was able to authenticate or sync. If an OAuth refresh token expires or
is revoked, the container log includes a reauthorization command, for example:

```text
Account 1: OAuth2 authorization is expired, revoked, or rejected by office365. Reauthorize with: docker compose run --rm imapsync authorize 1
```

After completing the browser authorization, start that account immediately with:

```bash
docker compose exec imapsync sync.sh sync 1
```

```bash
# Check health status
docker inspect --format='{{.State.Health.Status}}' email-sync-oauth2

# View healthcheck logs (useful for debugging)
docker inspect --format='{{json .State.Health}}' email-sync-oauth2 | jq

# Watch live heartbeat
docker exec email-sync-oauth2 cat /var/log/imapsync/.heartbeat
```

### Log files

```
./logs/
├── .heartbeat            # Healthcheck timestamp (updated every 60s)
├── account1.log          # Full imapsync output for account 1
├── account1.last_success # Unix timestamp of last successful sync
├── account1.last_failure # Unix timestamp of last failed sync
├── account2.log
├── notification.log      # SMTP notification delivery output
├── .notification_failed  # Present only while notification delivery is failing
└── ...
```

### Check sync status

```bash
# When did account 1 last succeed?
date -d @"$(cat logs/account1.last_success)"

# When did it last fail?
date -d @"$(cat logs/account1.last_failure)"
```

### Reauthorize reminders

For personal Microsoft accounts (90-day token limit), set a calendar reminder or monitoring check:

```bash
# Check if reauthorization is needed (token file older than 80 days)
find ./tokens -name "*.token" -mtime +80 -ls
```

---

## Architecture

```
┌─────────────────────────────────────────────────────┐
│                   Docker Container                  │
│                                                     │
│  ┌──────────────┐     ┌──────────────────────────┐ │
│  │   sync.sh    │────▶│   imapsync (per account) │ │
│  │  (daemon)    │     └──────────┬───────────────┘ │
│  │              │                │                  │
│  │  Interval:   │     ┌──────────▼───────────────┐ │
│  │  6h default  │     │   OAuth2 token helpers   │ │
│  │              │     │  oauth2_office365_with_  │ │
│  │  Healthcheck │     │  imap / oauth2_imap     │ │
│  └──────────────┘     └──────────┬───────────────┘ │
│                                   │                  │
│                        ┌──────────▼───────────────┐ │
│                        │   /tokens/ (mounted)     │ │
│                        │   account1.token         │ │
│                        │   account2.token         │ │
│                        └──────────────────────────┘ │
│                                                     │
│  ┌────────────────────────────────────────────────┐ │
│  │ /var/log/imapsync/ (mounted)                   │ │
│  │   account1.log, account1.last_success, etc.    │ │
│  └────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────┘

       │                                    │
       ▼                                    ▼
┌──────────────┐                    ┌──────────────┐
│  Source IMAP │                    │ Dest IMAP    │
│  (OAuth2)    │                    │ (password)   │
│  outlook.    │                    │ mail.hoelee  │
│  office365   │                    │ .com         │
└──────────────┘                    └──────────────┘
```

---

## FAQ

**Q: Do I need to reauthorize every 90 days?**
A: Only for personal Microsoft accounts (hotmail.com, outlook.com, live.com). Work/school Microsoft 365 accounts can have indefinite tokens. Gmail tokens last 7-90 days depending on app verification status.

**Q: Can I use this for a one-time migration?**
A: Yes! Set `ONCE=true` in `.env` and run `docker compose up`.

**Q: Is the sync incremental?**
A: Yes. imapsync only transfers emails that don't already exist on the destination.

**Q: Can I sync from any source to any destination?**
A: Any IMAP-compatible server. The source needs OAuth2 if it's Microsoft/Google. The destination always uses password auth.

**Q: How many accounts can I sync?**
A: Unlimited. Just increase `NUM_ACCOUNTS` and add `ACCOUNT_N_*` variables.

**Q: Can I use this with Synology DSM?**
A: Yes. Point Docker volume paths to `/volume1/docker/email-sync-oauth2/`.

**Q: What if my source is Exchange with ActiveSync?**
A: ActiveSync doesn't work well with mailcow (Microsoft disabled basic auth for ActiveSync). Use IMAP with OAuth2 instead.

---

## Building from Source

```bash
docker build -t email-sync-oauth2:latest .

docker run -d \
  --name email-sync-oauth2 \
  --restart unless-stopped \
  -e NUM_ACCOUNTS=1 \
  -e SYNC_INTERVAL=21600 \
  -e ACCOUNT_1_SOURCE_HOST=outlook.office365.com \
  -e ACCOUNT_1_SOURCE_USER=[email protected] \
  -e ACCOUNT_1_SOURCE_AUTH=oauth2 \
  -e ACCOUNT_1_SOURCE_PROVIDER=office365 \
  -e ACCOUNT_1_SOURCE_TOKEN=/tokens/you_hotmail.token \
  -e ACCOUNT_1_DEST_HOST=mail.hoelee.com \
  -e ACCOUNT_1_DEST_USER=[email protected] \
  -e ACCOUNT_1_DEST_PASS=yourpassword \
  -v $(pwd)/tokens:/tokens \
  -v $(pwd)/logs:/var/log/imapsync \
  email-sync-oauth2:latest
```

---

## License

MIT License. See [LICENSE](LICENSE) for details.

## Credits

- [imapsync](https://imapsync.lamiral.info/) by Gilles Lamiral — the core migration engine
- [imapsync OAuth2 helper](https://github.com/imapsync/imapsync/tree/master/oauth2/oauth2_imap) — OAuth2 authorization and refresh

---

## Contributing

Pull requests welcome! Please test with your own accounts before submitting.

1. Fork the repo
2. Create your feature branch (`git checkout -b feature/amazing-feature`)
3. Commit your changes (`git commit -m 'Add amazing feature'`)
4. Push to the branch (`git push origin feature/amazing-feature`)
5. Open a Pull Request