# Building a Background Gmail Sync Server with Temporal - Complete Tutorial

**Published by:** [Untitled](https://paragraph.com/@0x97498f957c7f33e3601d411cfdb806ff9c8b60bc/)
**Published on:** 2025-05-25
**URL:** https://paragraph.com/@0x97498f957c7f33e3601d411cfdb806ff9c8b60bc/building-a-background-gmail-sync-server-with-temporal-complete-tutorial

## Content

Gmail Automation Table of Contents What is Temporal? Basic Architecture Senior Staff Engineer Design for Small Startup/MVP Implementation Deployment & Cost Optimization Client Connection & Job Management Monitoring & Operations What is Temporal? Temporal is a durable execution platform that enables developers to write fault-tolerant applications that can survive failures, restarts, and infrastructure issues. Temporal was not an overnight idea. It took at least 15 years to get to where we are. Key Concepts Workflows: With Temporal, those steps are defined by writing code, known as a Workflow Definition, and are carried out by running that code, which results in a Workflow Execution. Workflows are resilient functions that maintain their state even across failures. Activities: Individual units of work (like API calls, database operations) that can fail and be retried independently. Workers: Workers are compute nodes that run your Temporal application code. You compile your workflows and activities into a worker executable. Task Queues: Communication mechanism between the Temporal service and workers. Why Temporal for Gmail Sync? Automatic retries: Handle Gmail API rate limits and network failures gracefully Long-running operations: Temporal Workflows are resilient. They can run—and keeping running—for years, even if the underlying infrastructure fails. State management: Track sync progress, handle partial syncs Visibility: Built-in UI to monitor and debug sync operations Basic Architecture Core Components ┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐ │ Client App │────▶│ Temporal Service │◀────│ Workers │ │ (API/Web/CLI) │ │ (Cloud/Self) │ │ (Your Code) │ └─────────────────┘ └──────────────────┘ └─────────────────┘ │ ▼ ┌─────────────────┐ │ Data Store │ │ (MySQL/Cassandra)│ └─────────────────┘ Gmail Sync Architecture ┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐ │ REST API │────▶│ Temporal Client │────▶│ Temporal Cloud │ │ (Express.js) │ │ │ │ │ └─────────────────┘ └──────────────────┘ └─────────────────┘ │ ▼ ┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐ │ Gmail API │◀────│ Sync Activities │◀────│ Sync Workflow │ │ │ │ │ │ │ └─────────────────┘ └──────────────────┘ └─────────────────┘ │ ▼ ┌─────────────────┐ │ PostgreSQL │ │ (Message Store) │ └─────────────────┘ Senior Staff Engineer Design for Small Startup/MVP Design Principles Simple & Extensible: Start with core sync functionality, design for future features Cost-Effective: Minimize infrastructure costs while maintaining reliability Observable: Built-in monitoring and debugging capabilities Scalable: Handle growth from 10 to 10,000 users without major refactoring System Design // Domain Model interface User { id: string; email: string; gmailRefreshToken: string; lastSyncHistoryId?: string; syncSettings: SyncSettings; } interface SyncSettings { syncFrequency: 'realtime' | 'hourly' | 'daily'; labelsToSync: string[]; maxMessagesPerSync: number; } interface SyncResult { messagesAdded: number; messagesModified: number; messagesDeleted: number; newHistoryId: string; errors: SyncError[]; } Workflow Design // Main Sync Workflow export async function userGmailSyncWorkflow( userId: string ): Promise<void> { // Long-running workflow that manages all sync operations for a user // 1. Initial full sync if needed const user = await activities.getUser(userId); if (!user.lastSyncHistoryId) { await activities.performFullSync(userId); } // 2. Set up recurring sync based on user preferences while (true) { try { // Check for Gmail push notifications or poll const hasChanges = await activities.checkForChanges(userId); if (hasChanges) { const syncResult = await activities.performIncrementalSync(userId); // Handle sync results if (syncResult.errors.length > 0) { await activities.notifyUser(userId, syncResult.errors); } } // Sleep based on sync frequency const sleepDuration = getSleepDuration(user.syncSettings.syncFrequency); await sleep(sleepDuration); } catch (error) { // Temporal will automatically retry based on retry policy console.error('Sync error:', error); await sleep('5m'); // Back off on errors } } } Implementation Project Structure gmail-sync-server/ ├── src/ │ ├── workflows/ │ │ ├── gmail-sync.workflow.ts │ │ └── index.ts │ ├── activities/ │ │ ├── gmail.activities.ts │ │ ├── database.activities.ts │ │ └── index.ts │ ├── worker/ │ │ └── index.ts │ ├── api/ │ │ ├── server.ts │ │ └── routes/ │ ├── client/ │ │ └── temporal.client.ts │ └── shared/ │ └── types.ts ├── docker-compose.yml ├── package.json └── tsconfig.json 1. Setting Up the Project # Create project mkdir gmail-sync-server cd gmail-sync-server npm init -y # Install dependencies npm install @temporalio/client @temporalio/worker @temporalio/workflow @temporalio/activity @temporalio/common npm install express @types/express npm install googleapis @types/node npm install pg @types/pg npm install --save-dev typescript @types/node ts-node nodemon 2. Gmail Integration Activities // src/activities/gmail.activities.ts import { google } from 'googleapis'; import { OAuth2Client } from 'google-auth-library'; export async function getGmailClient(refreshToken: string): Promise<any> { const oauth2Client = new OAuth2Client( process.env.GMAIL_CLIENT_ID, process.env.GMAIL_CLIENT_SECRET, process.env.GMAIL_REDIRECT_URL ); oauth2Client.setCredentials({ refresh_token: refreshToken }); return google.gmail({ version: 'v1', auth: oauth2Client }); } export async function performFullSync(userId: string): Promise<SyncResult> { const user = await getUserFromDB(userId); const gmail = await getGmailClient(user.gmailRefreshToken); const result: SyncResult = { messagesAdded: 0, messagesModified: 0, messagesDeleted: 0, newHistoryId: '', errors: [] }; try { // Call messages.list to retrieve the first page of message IDs. let pageToken: string | undefined; do { const response = await gmail.users.messages.list({ userId: 'me', maxResults: 100, pageToken, labelIds: user.syncSettings.labelsToSync }); if (response.data.messages) { // Process messages in batches const messageIds = response.data.messages.map(m => m.id); await processBatchMessages(gmail, userId, messageIds); result.messagesAdded += messageIds.length; } pageToken = response.data.nextPageToken; // Store the historyId for future incremental syncs if (response.data.historyId) { result.newHistoryId = response.data.historyId; } } while (pageToken && result.messagesAdded < user.syncSettings.maxMessagesPerSync); // Update user's lastSyncHistoryId await updateUserSyncState(userId, result.newHistoryId); } catch (error) { result.errors.push({ type: 'FULL_SYNC_ERROR', message: error.message, timestamp: new Date() }); } return result; } export async function performIncrementalSync(userId: string): Promise<SyncResult> { const user = await getUserFromDB(userId); const gmail = await getGmailClient(user.gmailRefreshToken); const result: SyncResult = { messagesAdded: 0, messagesModified: 0, messagesDeleted: 0, newHistoryId: user.lastSyncHistoryId || '', errors: [] }; try { // you can perform a partial sync using the history.list method to return all history records newer than the startHistoryId you specify in your request. const historyResponse = await gmail.users.history.list({ userId: 'me', startHistoryId: user.lastSyncHistoryId, labelId: user.syncSettings.labelsToSync }); if (historyResponse.data.history) { for (const historyRecord of historyResponse.data.history) { // Process each history record if (historyRecord.messagesAdded) { result.messagesAdded += historyRecord.messagesAdded.length; } if (historyRecord.messagesDeleted) { result.messagesDeleted += historyRecord.messagesDeleted.length; } if (historyRecord.labelsAdded || historyRecord.labelsRemoved) { result.messagesModified += 1; } } } if (historyResponse.data.historyId) { result.newHistoryId = historyResponse.data.historyId; await updateUserSyncState(userId, result.newHistoryId); } } catch (error) { if (error.code === 404) { // If the startHistoryId supplied by your client is outside the available range of history records, the API returns an HTTP 404 error response. // Need to perform full sync return performFullSync(userId); } result.errors.push({ type: 'INCREMENTAL_SYNC_ERROR', message: error.message, timestamp: new Date() }); } return result; } 3. Workflow Implementation // src/workflows/gmail-sync.workflow.ts import * as workflow from '@temporalio/workflow'; import type * as activities from '../activities'; // Import activity types const { performFullSync, performIncrementalSync, checkForChanges, getUser, notifyUser } = workflow.proxyActivities<typeof activities>({ startToCloseTimeout: '5m', retry: { initialInterval: '30s', backoffCoefficient: 2, maximumInterval: '10m', maximumAttempts: 5 } }); // Define signals for external control export const pauseSyncSignal = workflow.defineSignal('pauseSync'); export const resumeSyncSignal = workflow.defineSignal('resumeSync'); export const updateSettingsSignal = workflow.defineSignal<[SyncSettings]>('updateSettings'); export async function userGmailSyncWorkflow( userId: string ): Promise<void> { let isPaused = false; let syncSettings: SyncSettings | null = null; // Set up signal handlers workflow.setHandler(pauseSyncSignal, () => { isPaused = true; }); workflow.setHandler(resumeSyncSignal, () => { isPaused = false; }); workflow.setHandler(updateSettingsSignal, (newSettings) => { syncSettings = newSettings; }); // Main sync loop while (true) { if (isPaused) { await workflow.sleep('1m'); continue; } try { const user = await getUser(userId); if (!user) { throw new Error(`User ${userId} not found`); } // Use latest settings if (syncSettings) { user.syncSettings = syncSettings; } // Perform initial full sync if needed if (!user.lastSyncHistoryId) { await performFullSync(userId); } // Check for changes const hasChanges = await checkForChanges(userId); if (hasChanges) { const syncResult = await performIncrementalSync(userId); if (syncResult.errors.length > 0) { await notifyUser(userId, syncResult.errors); } } // Sleep based on sync frequency const sleepDuration = getSleepDuration(user.syncSettings.syncFrequency); await workflow.sleep(sleepDuration); } catch (error) { console.error(`Sync error for user ${userId}:`, error); // Temporal will retry based on the retry policy await workflow.sleep('5m'); // Back off on errors } } } function getSleepDuration(frequency: string): string { switch (frequency) { case 'realtime': return '1m'; case 'hourly': return '1h'; case 'daily': return '24h'; default: return '1h'; } } 4. Worker Implementation // src/worker/index.ts import { Worker, NativeConnection } from '@temporalio/worker'; import * as activities from '../activities'; async function run() { // Your first development environment will probably only need one worker that runs both workflows and activities in a single process. const connection = await NativeConnection.connect({ address: process.env.TEMPORAL_ADDRESS || 'localhost:7233' }); const worker = await Worker.create({ connection, namespace: process.env.TEMPORAL_NAMESPACE || 'default', taskQueue: 'gmail-sync-queue', workflowsPath: require.resolve('../workflows'), activities, }); await worker.run(); } run().catch((err) => { console.error(err); process.exit(1); }); 5. Client API Implementation // src/api/server.ts import express from 'express'; import { Connection, Client } from '@temporalio/client'; import { userGmailSyncWorkflow } from '../workflows/gmail-sync.workflow'; const app = express(); app.use(express.json()); let temporalClient: Client; async function getTemporalClient(): Promise<Client> { if (!temporalClient) { const connection = await Connection.connect({ address: process.env.TEMPORAL_ADDRESS || 'localhost:7233' }); temporalClient = new Client({ connection, namespace: process.env.TEMPORAL_NAMESPACE || 'default' }); } return temporalClient; } // Start sync for a user app.post('/api/users/:userId/sync/start', async (req, res) => { try { const { userId } = req.params; const client = await getTemporalClient(); // Although it is not required, we recommend providing your own Workflow Idthat maps to a business process or business entity identifier, such as an order identifier or customer identifier. const workflowId = `gmail-sync-${userId}`; const handle = await client.workflow.start(userGmailSyncWorkflow, { workflowId, taskQueue: 'gmail-sync-queue', args: [userId], // Workflow should run indefinitely workflowExecutionTimeout: undefined, }); res.json({ success: true, workflowId: handle.workflowId, runId: handle.firstExecutionRunId }); } catch (error) { res.status(500).json({ error: error.message }); } }); // Pause sync app.post('/api/users/:userId/sync/pause', async (req, res) => { try { const { userId } = req.params; const client = await getTemporalClient(); const workflowId = `gmail-sync-${userId}`; const handle = client.workflow.getHandle(workflowId); await handle.signal('pauseSync'); res.json({ success: true, message: 'Sync paused' }); } catch (error) { res.status(500).json({ error: error.message }); } }); // Update sync settings app.put('/api/users/:userId/sync/settings', async (req, res) => { try { const { userId } = req.params; const settings = req.body; const client = await getTemporalClient(); const workflowId = `gmail-sync-${userId}`; const handle = client.workflow.getHandle(workflowId); await handle.signal('updateSettings', settings); res.json({ success: true, settings }); } catch (error) { res.status(500).json({ error: error.message }); } }); // Check sync status app.get('/api/users/:userId/sync/status', async (req, res) => { try { const { userId } = req.params; const client = await getTemporalClient(); const workflowId = `gmail-sync-${userId}`; const handle = client.workflow.getHandle(workflowId); const description = await handle.describe(); res.json({ status: description.status.name, startTime: description.startTime, runId: description.runId }); } catch (error) { res.status(500).json({ error: error.message }); } }); const PORT = process.env.PORT || 3000; app.listen(PORT, () => { console.log(`API server listening on port ${PORT}`); }); Deployment & Cost Optimization Deployment Architecture Since Temporal: Workflows run on workers that live on compute that you manage. This could mean a server you own and operate, a Kubernetes cluster hosted in EKS or your data center, or even ECS Fargate., we'll use ECS Fargate for a cost-effective deployment. Cost Comparison Option 1: Temporal Cloud (Recommended for MVP) 🎉 Startup Programs Available: General Startup Program: Our startup program offers $6,000 in free Temporal Cloud credits to startups with less than $30M in funding. AWS Activate Members: Temporal has partnered with AWS Activate to provide $3,000 in free Temporal Cloud credits Google Cloud Startups: This exclusive offer for members of the Google for Startups Cloud Program (GFSCP) provides $12,000 in free Temporal Cloud credits over 12 months New Users: $1,000 in free credits for new self-service sign-ups Pricing Model: Actions pricing starts at $50 per million Actions ($0.00005 per Action). You gain progressive volume discounts as you scale. Estimated Costs for Gmail Sync: 1,000 users × 24 syncs/day × 30 days = 720,000 syncs/month Each sync ≈ 10 actions (workflow start, activities, timers) Total: ~7.2M actions/month = $360/month Storage: storage is generally a very small percentage (3-5%) of the total bill. Support: the minimum support fee is $200 per month. Total: ~$600/month for 1,000 active users (potentially covered by startup credits for 10+ months!) Option 2: Self-Hosted on AWS ECS Infrastructure Needs: # Minimal Production Setup services: temporal-frontend: cpu: 512 memory: 1024 desiredCount: 2 temporal-history: cpu: 1024 memory: 2048 desiredCount: 2 temporal-matching: cpu: 512 memory: 1024 desiredCount: 2 temporal-worker-service: cpu: 512 memory: 1024 desiredCount: 1 # Your workers gmail-sync-workers: cpu: 1024 memory: 2048 desiredCount: 2-10 # Auto-scale based on load Estimated Costs: ECS Fargate: ~$300-500/month RDS MySQL (db.t3.small): ~$30/month ALB: ~$20/month Data transfer: ~$50/month Total: ~$400-600/month Deployment Guide for ECS Fargate 1. Create ECS Task Definitions // temporal-worker-task-definition.json { "family": "gmail-sync-worker", "networkMode": "awsvpc", "requiresCompatibilities": ["FARGATE"], "cpu": "1024", "memory": "2048", "containerDefinitions": [ { "name": "worker", "image": "your-ecr-repo/gmail-sync-worker:latest", "environment": [ { "name": "TEMPORAL_ADDRESS", "value": "temporal-cloud-address:7233" }, { "name": "TEMPORAL_NAMESPACE", "value": "default" }, { "name": "DATABASE_URL", "value": "postgresql://..." } ], "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-group": "/ecs/gmail-sync-worker", "awslogs-region": "us-east-1", "awslogs-stream-prefix": "ecs" } } } ] } 2. Cost Optimization Strategies Use Fargate Spot for Workers: For lower level environments, such as test, engineering, or whatever your nomenclature is, it might not make sense to have tasks running all the time. // ECS Service configuration { "capacityProviderStrategy": [ { "capacityProvider": "FARGATE_SPOT", "weight": 80, "base": 0 }, { "capacityProvider": "FARGATE", "weight": 20, "base": 2 } ] } Auto-scaling Configuration: // Auto-scaling policy { "targetTrackingScalingPolicies": [ { "targetValue": 70.0, "predefinedMetricSpecification": { "predefinedMetricType": "ECSServiceAverageCPUUtilization" }, "scaleOutCooldown": 60, "scaleInCooldown": 300 } ] } Schedule-based Scaling: // Scale down during off-hours { "scheduledActions": [ { "scheduledActionName": "scale-down-night", "schedule": "cron(0 22 * * ? *)", "scalableTargetAction": { "minCapacity": 1, "maxCapacity": 2 } }, { "scheduledActionName": "scale-up-morning", "schedule": "cron(0 6 * * ? *)", "scalableTargetAction": { "minCapacity": 2, "maxCapacity": 10 } } ] } Docker Configuration # Dockerfile FROM node:18-alpine WORKDIR /app # Install dependencies COPY package*.json ./ RUN npm ci --only=production # Copy application code COPY . . # Build TypeScript RUN npm run build # Run the worker CMD ["node", "dist/worker/index.js"] Client Connection & Job Management Client SDK Usage // Example: React Web App import { useEffect, useState } from 'react'; function GmailSyncManager({ userId }) { const [syncStatus, setSyncStatus] = useState(null); const startSync = async () => { const response = await fetch(`/api/users/${userId}/sync/start`, { method: 'POST' }); const data = await response.json(); setSyncStatus(data); }; const pauseSync = async () => { await fetch(`/api/users/${userId}/sync/pause`, { method: 'POST' }); }; const updateSettings = async (settings) => { await fetch(`/api/users/${userId}/sync/settings`, { method: 'PUT', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify(settings) }); }; return ( <div> <button onClick={startSync}>Start Sync</button> <button onClick={pauseSync}>Pause Sync</button> {/* Sync settings UI */} </div> ); } Webhook Integration // Handle Gmail push notifications app.post('/api/webhooks/gmail', async (req, res) => { const { emailAddress, historyId } = req.body; // Find user by email const user = await findUserByEmail(emailAddress); if (user) { const client = await getTemporalClient(); const workflowId = `gmail-sync-${user.id}`; try { const handle = client.workflow.getHandle(workflowId); // Signal the workflow that changes are available await handle.signal('gmailChangesAvailable', { historyId }); } catch (error) { // Workflow might not be running, start it await client.workflow.start(userGmailSyncWorkflow, { workflowId, taskQueue: 'gmail-sync-queue', args: [user.id] }); } } res.status(200).send(); }); Monitoring & Operations Observability Setup // Add OpenTelemetry tracing import { Resource } from '@opentelemetry/resources'; import { SemanticResourceAttributes } from '@opentelemetry/semantic-conventions'; import { OpenTelemetryActivityInboundInterceptor } from '@temporalio/interceptors-opentelemetry'; const resource = Resource.default().merge( new Resource({ [SemanticResourceAttributes.SERVICE_NAME]: 'gmail-sync-worker', [SemanticResourceAttributes.SERVICE_VERSION]: process.env.SERVICE_VERSION, }) ); // Configure worker with interceptors const worker = await Worker.create({ // ... other config interceptors: { activity: [ new OpenTelemetryActivityInboundInterceptor() ] } }); Metrics & Alerting // CloudWatch metrics class MetricsCollector { async recordSyncMetrics(userId: string, result: SyncResult) { await cloudWatch.putMetricData({ Namespace: 'GmailSync', MetricData: [ { MetricName: 'MessagesAdded', Value: result.messagesAdded, Dimensions: [{ Name: 'UserId', Value: userId }] }, { MetricName: 'SyncErrors', Value: result.errors.length, Dimensions: [{ Name: 'UserId', Value: userId }] } ] }).promise(); } } Production Checklist Security: Store Gmail refresh tokens encrypted Use AWS Secrets Manager for credentials Enable TLS for Temporal connections Implement proper OAuth2 flow Reliability: Set up dead letter queues Implement circuit breakers for Gmail API Configure appropriate retry policies Set up health checks Performance: Batch Gmail API requests Implement message deduplication Use database connection pooling Cache frequently accessed data Cost Monitoring: Set up AWS Cost Explorer alerts Monitor Temporal Cloud usage Track Gmail API quota usage Implement usage-based billing for customers Conclusion This architecture provides a robust, scalable, and cost-effective solution for Gmail synchronization using Temporal. The key benefits include: Reliability: Automatic failure handling and retries Scalability: Easy to scale workers based on load Visibility: Built-in workflow tracking and debugging Cost-effective: Pay only for what you use with Temporal Cloud Maintainable: Clear separation of concerns with workflows and activities For a startup MVP, we can consider starting with Temporal Cloud to minimize operational overhead, then considering self-hosting once we reach ~5,000 active users or need more control over the infrastructure.

## Publication Information

- [Untitled](https://paragraph.com/@0x97498f957c7f33e3601d411cfdb806ff9c8b60bc/): Publication homepage
- [All Posts](https://paragraph.com/@0x97498f957c7f33e3601d411cfdb806ff9c8b60bc/): More posts from this publication
- [RSS Feed](https://api.paragraph.com/blogs/rss/@0x97498f957c7f33e3601d411cfdb806ff9c8b60bc): Subscribe to updates