Zimbra Issues

Andy Phillips

Andy Phillips

We recently upgraded to Zimbra 8.6 from 8.0.7. We hit a problem that not even zimbra support was able to figure out. Perhaps its that our Zimbra instance has been upgraded since version 3 (or 4) but it got into a bad state whenever we tried to update the proxy. Here’s the solution.

First the problem;

$ zmproxyctl status
zmnginxctl is not running
$ zmproxyctl restart
Stopping nginx...nginx is not running.
Starting nginx...nginx: [emerg] invalid port in upstream "mail.example.com:0" 
in /opt/zimbra/conf/nginx/includes/nginx.conf.web:62
failed.

So that’s fairly easy to find. 

Line 62 in conf/nginx/includes/nginx.conf.web

60 upstream zimbra_adminclient
61 {
62 server mail.example.com:0 fail_timeout=60s version=8.6.0_GA_1153;
63 zmauth_admin;
64 }
65 # Define the collection of upstream admin console servers to which we will
66 # proxy. Define each server:port against a server directive
67 #
68 upstream zimbra_admin
69 {
70 server mail.example.com:0 fail_timeout=60s version=8.6.0_GA_1153;
71 zmauth_admin;
72 }

The problem is the :0 

Nginx doesn’t like that as a port specification. 

Looking at the template – in this case nginx.conf.web.template

 upstream ${web.admin.upstream.name}
{
${web.admin.upstream.:servers}
zmauth_admin;
}

So what defines that. A google gives me this;

https://wiki.zimbra.com/wiki/Zimbra_Proxy…

which says;

 NGINX Keyword:         web.admin.upstream.:servers
 Description:           List of upstream admin servers used by Web Proxy 
(i.e. admin servers for which zimbraReverseProxyLookupTarget is true, and whose
mail mode is http|mixed|both) Controlling Attribute: zimbraReverseProxyLookupTarget Default Value: [] Current Value: [<server>:7071] Config Text: server <server>:7071; How to modify: N/A

Not exactly helpful. So we called Zimbra support, who suggested that we

  • Run zmproxyconfig (opt/zimbra/libexec/zmproxyconfig -e -o -w -m -C -n 7071:9071 -H `zmhostname`) – which takes down the server, as nginx fails to start. 
  • Remove all files in conf/nginx/include and re-running zmproxyconfgen – this also did not help

Digging into zmproxyconfgen arguments we find the -D flag.

-D' />,--definitions          Print variable map Definitions after loading
LDAP configuration (and processing
overrides). -D requires -s upstream server.
If "-s upstream server" is not specified, it
just dumps the default varaible map

So, diving in and looking for our problem area;

libexec/zmproxyconfgen -D -s mail.example.com 
NGINX Keyword: web.admin.upstream.:servers
Description: List of upstream HTTPS Admin client servers used by Web Proxy
Value Type: CUSTOM
Controlling Attribute: (none)
Default Value: (none)
Current Value: [mail.example.com:0 fail_timeout=60s version=8.6.0_GA_1153]
Config Text: server mail.example.com:0 fail_timeout=60s version=8.6.0_GA_1153;

So there’s our :0 again.

And elsewhere in the same output (because we know it should be 7071) we find;

NGINX Keyword: web.admin.uport
Description: Admin console upstream port
Value Type: INTEGER
Controlling Attribute: zimbraAdminPort
Default Value: 7071
Current Value: 7071
Config Text: 7071

We can look for zimbraAdminPort in the output and compare it to a fresh install on a test server;

1) A working test server;

$ zmprov gacf | grep AdminPort 
zimbraAdminPort: 7071
zimbraReverseProxyAdminPortAttribute: zimbraAdminPort

2) Our problem server

$ zmprov -l gacf | grep  AdminPort 
zimbraAdminPort: 7071

In other words, we’re missing the zimbraReverseProxyAdminPortAttribute. 

Are there any other attributes we’re missing?

On the clean install;

$ zmprov gacf | grep Attribute 
zimbraContactHiddenAttributes: dn,vcardUID,vcardURL,vcardXProps,member
zimbraReverseProxyAdminPortAttribute: zimbraAdminPort
zimbraReverseProxyDomainNameAttribute: zimbraDomainName
zimbraReverseProxyHttpPortAttribute: zimbraMailPort
zimbraReverseProxyHttpSSLPortAttribute: zimbraMailSSLPort
zimbraReverseProxyImapPortAttribute: zimbraImapBindPort
zimbraReverseProxyImapSSLPortAttribute: zimbraImapSSLBindPort
zimbraReverseProxyMailHostAttribute: zimbraMailHost
zimbraReverseProxyPop3PortAttribute: zimbraPop3BindPort
zimbraReverseProxyPop3SSLPortAttribute: zimbraPop3SSLBindPort

On the problem server

[[email protected] ~]$ zmprov gacf | grep Attribute 
zimbraContactHiddenAttributes: dn,vcardUID,vcardURL,vcardXProps,member
zimbraReverseProxyDomainNameAttribute: zimbraDomainName
zimbraReverseProxyHttpPortAttribute: zimbraMailPort
zimbraReverseProxyHttpSSLPortAttribute: zimbraMailSSLPort
zimbraReverseProxyImapPortAttribute: zimbraImapBindPort
zimbraReverseProxyImapSSLPortAttribute: zimbraImapSSLBindPort
zimbraReverseProxyMailHostAttribute: zimbraMailHost
zimbraReverseProxyPop3PortAttribute: zimbraPop3BindPort
zimbraReverseProxyPop3SSLPortAttribute: zimbraPop3SSLBindPort

So, it looks like its just that one. 

Fixing it using;

$ zmprov -l mcf zimbraReverseProxyAdminPortAttribute zimbraAdminPort
$ zmprov -l gacf zimbraReverseProxyAdminPortAttribute
zimbraReverseProxyAdminPortAttribute: zimbraAdminPort

Regenerating config and checking;

$ libexec/zmproxyconfgen -D -s mail.example.com > /tmp/zmproxyconfgen.out2
$ diff -u /tmp/zmproxyconfgen.out /tmp/zmproxyconfgen.out2
--- /tmp/zmproxyconfgen.out 2016-02-09 12:30:42.856618559 +0000
+++ /tmp/zmproxyconfgen.out2 2016-02-09 12:42:37.146391380 +0000
@@ -607,8 +607,8 @@
Value Type: CUSTOM
Controlling Attribute: (none)
Default Value: (none)
- Current Value: [mail.example.com:0 fail_timeout=60s version=8.6.0_GA_1153]
- Config Text: server mail.example.com:0 fail_timeout=60s version=8.6.0_GA_1153;
+ Current Value: [mail.example.com:7071 fail_timeout=60s version=8.6.0_GA_1153]
+ Config Text: server mail.example.com:7071 fail_timeout=60s version=8.6.0_GA_1153;

NGINX Keyword: web.admin.upstream.:servers
@@ -616,8 +616,8 @@
Value Type: CUSTOM
Controlling Attribute: (none)
Default Value: (none)
- Current Value: [mail.example.com:0 fail_timeout=60s version=8.6.0_GA_1153]
- Config Text: server mail.example.com:0 fail_timeout=60s version=8.6.0_GA_1153;
+ Current Value: [mail.example.com:7071 fail_timeout=60s version=8.6.0_GA_1153]
+ Config Text: server mail.example.com:7071 fail_timeout=60s version=8.6.0_GA_1153;

now, looking again in conf/nginx/includes

upstream zimbra_adminclient
{
server mail.example.com:7071 fail_timeout=60s version=8.6.0_GA_1153;
zmauth_admin;
}
# Define the collection of upstream admin console servers to which we will
# proxy. Define each server:port against a server directive
#
upstream zimbra_admin
{
server mail.example.com:7071 fail_timeout=60s version=8.6.0_GA_1153;
zmauth_admin;
}

Problem fixed. nginx now starts. 

All we need to do now is figure out why imap completely fails when we have the proxy active. 

Hope that helped.

Any opinions, news, research, analyses, prices or other information ("information") contained on this Blog, constitutes marketing communication and it has not been prepared in accordance with legal requirements designed to promote the independence of investment research. Further, the information contained within this Blog does not contain (and should not be construed as containing) investment advice or an investment recommendation, or an offer of, or solicitation for, a transaction in any financial instrument. LMAX Exchange has not verified the accuracy or basis-in-fact of any claim or statement made by any third parties as comments for every Blog entry.

LMAX Exchange will not accept liability for any loss or damage, including without limitation to, any loss of profit, which may arise directly or indirectly from use of or reliance on such information. No representation or warranty is given as to the accuracy or completeness of the above information. While the produced information was obtained from sources deemed to be reliable, LMAX Exchange does not provide any guarantees about the reliability of such sources. Consequently any person acting on it does so entirely at his or her own risk. It is not a place to slander, use unacceptable language or to promote LMAX Exchange or any other FX, Spread Betting and CFD provider and any such postings, excessive or unjust comments and attacks will not be allowed and will be removed from the site immediately.

LMAX Exchange will clearly identify and mark any content it publishes or that is approved by LMAX Exchange.

FX and CFDs are leveraged products that can result in losses exceeding your deposit. They are not suitable for everyone so please ensure you fully understand the risks involved. The information on this website is not directed at residents of the United States of America, Australia (we will only deal with Australian clients who are "wholesale clients" as defined under the Corporations Act 2001), Canada (although we may deal with Canadian residents who meet the "Permitted Client" criteria), Singapore or any other jurisdiction where FX trading and/or CFD trading is restricted or prohibited by local laws or regulations.